lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-java7 - Build # 603 - Still Failing
Date Thu, 01 Nov 2012 10:09:47 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-java7/603/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=407, name=qtp680254087-407 Acceptor3 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=406, name=qtp680254087-406 Acceptor2 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=405, name=qtp680254087-405 Acceptor1 SelectChannelConnector@0.0.0.0:22157, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=338, name=qtp551926596-338 Acceptor2 SelectChannelConnector@0.0.0.0:57705, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=336, name=qtp551926596-336 Acceptor0 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=337, name=qtp551926596-337 Acceptor1 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=339, name=qtp551926596-339 Acceptor3 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=404, name=qtp680254087-404 Acceptor0 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=407, name=qtp680254087-407 Acceptor3 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=406, name=qtp680254087-406 Acceptor2 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=405, name=qtp680254087-405 Acceptor1 SelectChannelConnector@0.0.0.0:22157, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=338, name=qtp551926596-338 Acceptor2 SelectChannelConnector@0.0.0.0:57705, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=336, name=qtp551926596-336 Acceptor0 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=337, name=qtp551926596-337 Acceptor1 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=339, name=qtp551926596-339 Acceptor3 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=404, name=qtp680254087-404 Acceptor0 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([10754BD6AFFFB7CB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=407, name=qtp680254087-407 Acceptor3 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=406, name=qtp680254087-406 Acceptor2 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=405, name=qtp680254087-405 Acceptor1 SelectChannelConnector@0.0.0.0:22157, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=338, name=qtp551926596-338 Acceptor2 SelectChannelConnector@0.0.0.0:57705, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=336, name=qtp551926596-336 Acceptor0 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=337, name=qtp551926596-337 Acceptor1 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=339, name=qtp551926596-339 Acceptor3 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=404, name=qtp680254087-404 Acceptor0 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=407, name=qtp680254087-407 Acceptor3 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=406, name=qtp680254087-406 Acceptor2 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=405, name=qtp680254087-405 Acceptor1 SelectChannelConnector@0.0.0.0:22157, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=338, name=qtp551926596-338 Acceptor2 SelectChannelConnector@0.0.0.0:57705, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=336, name=qtp551926596-336 Acceptor0 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=337, name=qtp551926596-337 Acceptor1 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=339, name=qtp551926596-339 Acceptor3 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=404, name=qtp680254087-404 Acceptor0 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([10754BD6AFFFB7CB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=207, name=qtp930016431-207 Acceptor0 SelectChannelConnector@0.0.0.0:21225, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=210, name=qtp930016431-210 Acceptor3 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=300, name=qtp1564709036-300 Acceptor0 SelectChannelConnector@0.0.0.0:21235, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=227, name=qtp50250627-227 Acceptor2 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=226, name=qtp50250627-226 Acceptor1 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=208, name=qtp930016431-208 Acceptor1 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=225, name=qtp50250627-225 Acceptor0 SelectChannelConnector@0.0.0.0:21230, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=301, name=qtp1564709036-301 Acceptor1 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=209, name=qtp930016431-209 Acceptor2 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=228, name=qtp50250627-228 Acceptor3 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=302, name=qtp1564709036-302 Acceptor2 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=303, name=qtp1564709036-303 Acceptor3 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest: 
   1) Thread[id=207, name=qtp930016431-207 Acceptor0 SelectChannelConnector@0.0.0.0:21225, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=210, name=qtp930016431-210 Acceptor3 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=300, name=qtp1564709036-300 Acceptor0 SelectChannelConnector@0.0.0.0:21235, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=227, name=qtp50250627-227 Acceptor2 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=226, name=qtp50250627-226 Acceptor1 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=208, name=qtp930016431-208 Acceptor1 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=225, name=qtp50250627-225 Acceptor0 SelectChannelConnector@0.0.0.0:21230, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=301, name=qtp1564709036-301 Acceptor1 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=209, name=qtp930016431-209 Acceptor2 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=228, name=qtp50250627-228 Acceptor3 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=302, name=qtp1564709036-302 Acceptor2 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=303, name=qtp1564709036-303 Acceptor3 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([10754BD6AFFFB7CB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=301, name=qtp1564709036-301 Acceptor1 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=207, name=qtp930016431-207 Acceptor0 SelectChannelConnector@0.0.0.0:21225, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=210, name=qtp930016431-210 Acceptor3 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=300, name=qtp1564709036-300 Acceptor0 SelectChannelConnector@0.0.0.0:21235, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=227, name=qtp50250627-227 Acceptor2 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=226, name=qtp50250627-226 Acceptor1 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=209, name=qtp930016431-209 Acceptor2 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=228, name=qtp50250627-228 Acceptor3 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=302, name=qtp1564709036-302 Acceptor2 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=303, name=qtp1564709036-303 Acceptor3 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=208, name=qtp930016431-208 Acceptor1 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=225, name=qtp50250627-225 Acceptor0 SelectChannelConnector@0.0.0.0:21230, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=301, name=qtp1564709036-301 Acceptor1 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=207, name=qtp930016431-207 Acceptor0 SelectChannelConnector@0.0.0.0:21225, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=210, name=qtp930016431-210 Acceptor3 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=300, name=qtp1564709036-300 Acceptor0 SelectChannelConnector@0.0.0.0:21235, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=227, name=qtp50250627-227 Acceptor2 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=226, name=qtp50250627-226 Acceptor1 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=209, name=qtp930016431-209 Acceptor2 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=228, name=qtp50250627-228 Acceptor3 SelectChannelConnector@0.0.0.0:21230, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=302, name=qtp1564709036-302 Acceptor2 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=303, name=qtp1564709036-303 Acceptor3 SelectChannelConnector@0.0.0.0:21235, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=208, name=qtp930016431-208 Acceptor1 SelectChannelConnector@0.0.0.0:21225, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=225, name=qtp50250627-225 Acceptor0 SelectChannelConnector@0.0.0.0:21230, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([10754BD6AFFFB7CB]:0)




Build Log:
[...truncated 8368 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4]   2> 5 T191 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-RecoveryZkTest-1351763754330
[junit4:junit4]   2> 6 T191 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 10 T192 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 11 T192 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 12 T192 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 107 T191 oasc.ZkTestServer.run start zk server on port:21221
[junit4:junit4]   2> 107 T191 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:21221 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@16432f65
[junit4:junit4]   2> 108 T197 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:21221
[junit4:junit4]   2> 108 T191 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 109 T197 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:21221, initiating session
[junit4:junit4]   2> 110 T193 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21222
[junit4:junit4]   2> 110 T193 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21222
[junit4:junit4]   2> 111 T195 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 150 T195 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abb65fd610000 with negotiated timeout 10000 for client /140.211.11.196:21222
[junit4:junit4]   2> 150 T197 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:21221, sessionid = 0x13abb65fd610000, negotiated timeout = 10000
[junit4:junit4]   2> 151 T198 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16432f65 name:ZooKeeperConnection Watcher:127.0.0.1:21221 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 151 T191 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 152 T191 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 155 T196 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abb65fd610000
[junit4:junit4]   2> 157 T198 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 157 T191 oaz.ZooKeeper.close Session: 0x13abb65fd610000 closed
[junit4:junit4]   2> 157 T193 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:21222 which had sessionid 0x13abb65fd610000
[junit4:junit4]   2> 158 T191 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:21221/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@10fc040
[junit4:junit4]   2> 159 T199 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:21221
[junit4:junit4]   2> 159 T191 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 160 T199 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:21221, initiating session
[junit4:junit4]   2> 160 T193 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21223
[junit4:junit4]   2> 161 T193 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21223
[junit4:junit4]   2> 162 T195 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abb65fd610001 with negotiated timeout 10000 for client /140.211.11.196:21223
[junit4:junit4]   2> 162 T199 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:21221, sessionid = 0x13abb65fd610001, negotiated timeout = 10000
[junit4:junit4]   2> 163 T200 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10fc040 name:ZooKeeperConnection Watcher:127.0.0.1:21221/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 163 T191 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 164 T191 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 168 T191 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 172 T191 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 183 T191 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 187 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 208 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 222 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 223 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 327 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 328 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 339 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 340 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 350 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 351 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 423 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 424 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 429 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 430 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 478 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 480 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 484 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 485 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 526 T191 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 527 T191 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 551 T196 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abb65fd610001
[junit4:junit4]   2> 592 T200 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 592 T191 oaz.ZooKeeper.close Session: 0x13abb65fd610001 closed
[junit4:junit4]   2> 592 T193 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:21223 which had sessionid 0x13abb65fd610001
[junit4:junit4]   2> 1160 T191 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1435 T191 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:21225
[junit4:junit4]   2> 1464 T191 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1464 T191 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1465 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1500 T191 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1500 T191 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1501 T191 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1501 T191 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1501 T191 oasc.CoreContainer.<init> New CoreContainer 1184377691
[junit4:junit4]   2> 1502 T191 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1503 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1540 T191 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1555 T191 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:21221/solr
[junit4:junit4]   2> 1572 T191 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:21221/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@1e60a6ce
[junit4:junit4]   2> 1573 T211 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:21221
[junit4:junit4]   2> 1573 T191 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1575 T211 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:21221, initiating session
[junit4:junit4]   2> 1576 T193 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21226
[junit4:junit4]   2> 1576 T193 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21226
[junit4:junit4]   2> 1578 T195 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abb65fd610002 with negotiated timeout 5000 for client /140.211.11.196:21226
[junit4:junit4]   2> 1578 T211 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:21221, sessionid = 0x13abb65fd610002, negotiated timeout = 5000
[junit4:junit4]   2> 1579 T212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e60a6ce name:ZooKeeperConnection Watcher:127.0.0.1:21221/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1579 T191 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1582 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1596 T191 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1598 T191 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:21225_solr
[junit4:junit4]   2> 1599 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:21225_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:21225_solr
[junit4:junit4]   2> 1604 T191 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:21225_solr
[junit4:junit4]   2> 1609 T191 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1609 T191 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1610 T191 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1610 T191 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1610 T191 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1611 T191 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1611 T191 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1612 T191 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1612 T191 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1613 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1625 T191 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1634 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1643 T191 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1647 T191 oasc.Overseer.start Overseer (id=88589189404229634-127.0.0.1:21225_solr-n_0000000000) starting
[junit4:junit4]   2> 1647 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1655 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1657 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1661 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1668 T214 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1668 T191 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1671 T191 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1687 T213 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1716 T191 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1717 T191 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 1718 T191 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1718 T191 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 1720 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1722 T191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1722 T191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1835 T191 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1898 T191 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2000 T191 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2042 T191 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2808 T191 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2829 T191 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2834 T191 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2869 T191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2877 T191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2885 T191 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2890 T191 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2896 T191 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351763754329/control/data/
[junit4:junit4]   2> 2902 T191 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e2f10c2
[junit4:junit4]   2> 2944 T191 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351763754329/control/data/index/
[junit4:junit4]   2> 2946 T191 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351763754329/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2947 T191 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351763754329/control/data/index forceNew:false
[junit4:junit4]   2> 2959 T191 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351763754329/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2960 T191 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2962 T191 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2966 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2966 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2967 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2968 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2970 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2970 T191 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2971 T191 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2973 T191 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2974 T191 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2975 T191 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2977 T191 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2978 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2979 T191 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2980 T191 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2981 T191 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2982 T191 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2983 T191 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2985 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2986 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2987 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2989 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2990 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2992 T191 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2993 T191 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2995 T191 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 3011 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3017 T191 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 3019 T191 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 3026 T191 oass.SolrIndexSearcher.<init> Opening Searcher@4cde17e5 main
[junit4:junit4]   2> 3056 T191 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3057 T191 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3058 T191 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 3068 T191 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 3125 T219 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4cde17e5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3140 T191 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3141 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x56 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 3153 T191 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3154 T191 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:21225/solr shardId:control_shard
[junit4:junit4]   2> 3170 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 3192 T191 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 3204 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3206 T213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:21225_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:21225/solr"}
[junit4:junit4]   2> 3208 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:delete cxid:0x6a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 3217 T191 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 3218 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x6b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3222 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x6d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 3242 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3723 T191 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 3724 T191 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 3724 T191 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:21225/solr/collection1/
[junit4:junit4]   2> 3725 T191 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 3726 T191 oasc.SyncStrategy.syncToMe http://127.0.0.1:21225/solr/collection1/ has no replicas
[junit4:junit4]   2> 3727 T191 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:21225/solr/collection1/
[junit4:junit4]   2> 3727 T191 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 3734 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610002 type:create cxid:0x8b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3750 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3762 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3796 T191 oasc.ZkController.register We are http://127.0.0.1:21225/solr/collection1/ and leader is http://127.0.0.1:21225/solr/collection1/
[junit4:junit4]   2> 3800 T191 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:21225/solr
[junit4:junit4]   2> 3801 T191 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3802 T191 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3804 T191 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3806 T191 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3807 T191 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3879 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4109 T191 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 4113 T191 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:21230
[junit4:junit4]   2> 4114 T191 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4115 T191 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228
[junit4:junit4]   2> 4115 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228/'
[junit4:junit4]   2> 4168 T191 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4169 T191 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4169 T191 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228
[junit4:junit4]   2> 4170 T191 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228/solr.xml
[junit4:junit4]   2> 4171 T191 oasc.CoreContainer.<init> New CoreContainer 1232784122
[junit4:junit4]   2> 4172 T191 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228/'
[junit4:junit4]   2> 4172 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228/'
[junit4:junit4]   2> 4214 T191 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4235 T191 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:21221/solr
[junit4:junit4]   2> 4235 T191 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:21221/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@61c815f9
[junit4:junit4]   2> 4237 T229 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:21221
[junit4:junit4]   2> 4237 T191 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4238 T229 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:21221, initiating session
[junit4:junit4]   2> 4239 T193 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21231
[junit4:junit4]   2> 4239 T193 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21231
[junit4:junit4]   2> 4242 T195 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abb65fd610003 with negotiated timeout 5000 for client /140.211.11.196:21231
[junit4:junit4]   2> 4242 T229 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:21221, sessionid = 0x13abb65fd610003, negotiated timeout = 5000
[junit4:junit4]   2> 4243 T230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61c815f9 name:ZooKeeperConnection Watcher:127.0.0.1:21221/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4244 T191 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4245 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4247 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4253 T191 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:21230_solr
[junit4:junit4]   2> 4255 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:21230_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:21230_solr
[junit4:junit4]   2> 4256 T191 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:21230_solr
[junit4:junit4]   2> 4265 T212 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4266 T191 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4267 T191 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4267 T191 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4268 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4268 T191 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4269 T191 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4270 T213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:21225_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:21225/solr"}
[junit4:junit4]   2> 4269 T191 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4270 T191 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4271 T191 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4272 T191 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4273 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4284 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 4297 T191 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 4307 T191 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228/collection1
[junit4:junit4]   2> 4308 T191 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 4309 T191 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 4310 T191 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 4312 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228/collection1/'
[junit4:junit4]   2> 4314 T191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty1-1351763758228/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4315 T191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty1-1351763758228/collection1/lib/README' to classloader
[junit4:junit4]   2> 4473 T191 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4593 T191 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4695 T191 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4747 T191 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5660 T191 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5691 T191 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5708 T191 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5752 T191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5763 T191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5771 T191 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5775 T191 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5779 T191 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351763758228/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty1/
[junit4:junit4]   2> 5780 T191 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e2f10c2
[junit4:junit4]   2> 5782 T191 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty1/index/
[junit4:junit4]   2> 5783 T191 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5784 T191 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty1/index forceNew:false
[junit4:junit4]   2> 5792 T191 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty1/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5793 T191 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5794 T191 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 5799 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5799 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 5801 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5802 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 5803 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5804 T191 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5805 T191 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5806 T191 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5808 T191 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5809 T191 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 5811 T191 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 5812 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5813 T191 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 5814 T191 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 5815 T191 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 5816 T191 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5818 T191 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5819 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5821 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5822 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5823 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5825 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5826 T191 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5828 T191 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 5829 T191 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 5843 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5849 T191 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5851 T191 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 5858 T191 oass.SolrIndexSearcher.<init> Opening Searcher@310a7aa1 main
[junit4:junit4]   2> 5859 T191 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5860 T191 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5862 T191 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 5871 T191 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5944 T235 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@310a7aa1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5958 T191 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6299 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6300 T213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:21230_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:21230/solr"}
[junit4:junit4]   2> 6301 T213 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6318 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6318 T230 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6462 T191 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 6463 T191 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:21230/solr shardId:shard1
[junit4:junit4]   2> 6464 T191 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 6482 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610003 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 6483 T191 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 6484 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610003 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6493 T191 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 6494 T191 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 6495 T191 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:21230/solr/collection1/
[junit4:junit4]   2> 6495 T191 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 6496 T191 oasc.SyncStrategy.syncToMe http://127.0.0.1:21230/solr/collection1/ has no replicas
[junit4:junit4]   2> 6497 T191 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:21230/solr/collection1/
[junit4:junit4]   2> 6497 T191 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 6503 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610003 type:create cxid:0x4d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6837 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6864 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6864 T230 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6870 T191 oasc.ZkController.register We are http://127.0.0.1:21230/solr/collection1/ and leader is http://127.0.0.1:21230/solr/collection1/
[junit4:junit4]   2> 6870 T191 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:21230/solr
[junit4:junit4]   2> 6871 T191 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 6872 T191 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6876 T191 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6878 T191 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 6878 T191 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6879 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7092 T191 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 7096 T191 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:21235
[junit4:junit4]   2> 7097 T191 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7097 T191 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222
[junit4:junit4]   2> 7098 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222/'
[junit4:junit4]   2> 7149 T191 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 7150 T191 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7151 T191 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222
[junit4:junit4]   2> 7151 T191 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222/solr.xml
[junit4:junit4]   2> 7152 T191 oasc.CoreContainer.<init> New CoreContainer 1000606924
[junit4:junit4]   2> 7153 T191 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222/'
[junit4:junit4]   2> 7154 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222/'
[junit4:junit4]   2> 7193 T191 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 7214 T191 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:21221/solr
[junit4:junit4]   2> 7215 T191 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:21221/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@c32ddc5
[junit4:junit4]   2> 7216 T245 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:21221
[junit4:junit4]   2> 7216 T191 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7217 T245 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:21221, initiating session
[junit4:junit4]   2> 7217 T193 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21237
[junit4:junit4]   2> 7218 T193 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21237
[junit4:junit4]   2> 7221 T195 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abb65fd610004 with negotiated timeout 5000 for client /140.211.11.196:21237
[junit4:junit4]   2> 7221 T245 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:21221, sessionid = 0x13abb65fd610004, negotiated timeout = 5000
[junit4:junit4]   2> 7222 T246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c32ddc5 name:ZooKeeperConnection Watcher:127.0.0.1:21221/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7222 T191 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7223 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7225 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7236 T191 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:21235_solr
[junit4:junit4]   2> 7237 T196 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abb65fd610004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:21235_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:21235_solr
[junit4:junit4]   2> 7239 T191 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:21235_solr
[junit4:junit4]   2> 7243 T212 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 7243 T230 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 7244 T191 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7244 T191 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7245 T191 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7246 T191 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7246 T191 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7247 T191 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7247 T191 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7248 T191 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7249 T191 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7249 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 7273 T191 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7282 T191 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222/collection1
[junit4:junit4]   2> 7282 T191 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 7284 T191 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 7284 T191 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 7286 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222/collection1/'
[junit4:junit4]   2> 7288 T191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty2-1351763761222/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 7289 T191 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty2-1351763761222/collection1/lib/README' to classloader
[junit4:junit4]   2> 7372 T191 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 7378 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7379 T213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:21230_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:21230/solr"}
[junit4:junit4]   2> 7393 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7393 T230 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7393 T246 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7464 T191 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 7565 T191 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 7604 T191 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 8385 T191 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 8399 T191 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 8402 T191 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 8424 T191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8430 T191 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8435 T191 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8438 T191 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8441 T191 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351763761222/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty2/
[junit4:junit4]   2> 8441 T191 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2e2f10c2
[junit4:junit4]   2> 8443 T191 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty2/index/
[junit4:junit4]   2> 8443 T191 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 8444 T191 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty2/index forceNew:false
[junit4:junit4]   2> 8450 T191 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty2/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 8451 T191 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 8452 T191 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 8456 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 8457 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 8458 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 8458 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 8460 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 8461 T191 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 8462 T191 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 8463 T191 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 8465 T191 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 8466 T191 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 8467 T191 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 8468 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 8470 T191 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 8470 T191 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 8471 T191 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 8472 T191 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 8474 T191 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8475 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8476 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8478 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8479 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8480 T191 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8481 T191 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8482 T191 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 8483 T191 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 8492 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 8496 T191 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 8498 T191 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 8501 T191 oass.SolrIndexSearcher.<init> Opening Searcher@72132192 main
[junit4:junit4]   2> 8502 T191 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 8503 T191 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 8504 T191 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 8512 T191 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 8561 T250 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72132192 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 8570 T191 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8905 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8906 T213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:21235_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:21235/solr"}
[junit4:junit4]   2> 8925 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8925 T246 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8925 T230 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9073 T191 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 9073 T191 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:21235/solr shardId:shard1
[junit4:junit4]   2> 9077 T191 oasc.ZkController.register We are http://127.0.0.1:21235/solr/collection1/ and leader is http://127.0.0.1:21230/solr/collection1/
[junit4:junit4]   2> 9078 T191 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:21235/solr
[junit4:junit4]   2> 9078 T191 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 9079 T191 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@28e66cff url=http://127.0.0.1:21235/solr/collection1 node=127.0.0.1:21235_solr
[junit4:junit4]   2> 9085 T251 C2 P21235 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 9086 T251 C2 P21235 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 9086 T191 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9087 T251 C2 P21235 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 9087 T191 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 9088 T191 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 9089 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 9097 T251 C2 P21235 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:21230/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 9108 T251 C2 P21235 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 9112 T191 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:21221/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@487c05c8
[junit4:junit4]   2> 9113 T252 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:21221
[junit4:junit4]   2> 9113 T191 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 9114 T252 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:21221, initiating session
[junit4:junit4]   2> 9114 T193 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21240
[junit4:junit4]   2> 9115 T193 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21240
[junit4:junit4]   2> 9117 T195 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abb65fd610005 with negotiated timeout 10000 for client /140.211.11.196:21240
[junit4:junit4]   2> 9117 T252 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:21221, sessionid = 0x13abb65fd610005, negotiated timeout = 10000
[junit4:junit4]   2> 9117 T253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@487c05c8 name:ZooKeeperConnection Watcher:127.0.0.1:21221/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 9117 T191 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 9118 T191 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9123 T191 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 9124 T191 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 9130 T251 C2 P21235 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:21235/solr START replicas=[http://127.0.0.1:21230/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 9139 T191 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9150 T251 C2 P21235 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 9150 T251 C2 P21235 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 9151 T251 C2 P21235 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 9151 T251 C2 P21235 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 9431 T213 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9432 T213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:21235_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:21235/solr"}
[junit4:junit4]   2> 9446 T212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9446 T230 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9446 T253 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9446 T246 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@79b8e347 url=http://127.0.0.1:21230/solr/collection1 node=127.0.0.1:21230_solr
[junit4:junit4]   2> 10204 T233 C3 P21230 REQ /get {distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=20 
[junit4:junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@745fc59f url=http://127.0.0.1:21225/solr/collection1 node=127.0.0.1:21225_solr
[junit4:junit4]   2> 10263 T218 C4 P21225 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16
[junit4:junit4]   2> 10334 T216 C4 P21225 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351763754329/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10335 T216 C4 P21225 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10380 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10001 (1417427041282686976)]} 0 136
[junit4:junit4]   2> 10380 T218 C4 P21225 /update {wt=javabin&version=2} {add=[1 (1417427041282686977)]} 0 136
[junit4:junit4]   2> 10386 T255 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:21221/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@717ec56c
[junit4:junit4]   2> 10388 T257 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:21221
[junit4:junit4]   2> 10388 T255 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 10389 T257 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:21221, initiating session
[junit4:junit4]   2> 10389 T193 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21247
[junit4:junit4]   2> 10390 T193 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21247
[junit4:junit4]   2> 10445 T195 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abb65fd610006 with negotiated timeout 10000 for client /140.211.11.196:21247
[junit4:junit4]   2> 10445 T257 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:21221, sessionid = 0x13abb65fd610006, negotiated timeout = 10000
[junit4:junit4]   2> 10446 T258 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@717ec56c name:ZooKeeperConnection Watcher:127.0.0.1:21221/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 10446 T255 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 10448 T255 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 10481 T234 C3 P21230 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty1/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10482 T234 C3 P21230 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2>  C2_STATE=coll:collection1 core:collection1 props:{shard=shard1, roles=null, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:21235_solr, base_url=http://127.0.0.1:21235/solr}
[junit4:junit4]   2> 10552 T249 C2 P21235 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351763754329/jetty2/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10553 T249 C2 P21235 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10561 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=15 
[junit4:junit4]   2> 10563 T234 C3 P21230 /update {wt=javabin&version=2} {add=[1 (1417427041438924801)]} 0 96
[junit4:junit4]   2> 10565 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=19 
[junit4:junit4]   2> 10566 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10001 (1417427041438924800)]} 0 100
[junit4:junit4]   2> 10575 T217 C4 P21225 /update {wt=javabin&version=2} {add=[2 (1417427041537490944)]} 0 6
[junit4:junit4]   2> 10578 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10002 (1417427041539588096)]} 0 7
[junit4:junit4]   2> 10591 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10592 T232 C3 P21230 /update {wt=javabin&version=2} {add=[2 (1417427041547976704)]} 0 12
[junit4:junit4]   2> 10595 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10596 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10002 (1417427041551122432)]} 0 13
[junit4:junit4]   2> 10600 T218 C4 P21225 /update {wt=javabin&version=2} {add=[3 (1417427041565802496)]} 0 3
[junit4:junit4]   2> 10603 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10003 (1417427041568948224)]} 0 3
[junit4:junit4]   2> 10616 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10617 T234 C3 P21230 /update {wt=javabin&version=2} {add=[3 (1417427041574191104)]} 0 12
[junit4:junit4]   2> 10618 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10619 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10003 (1417427041577336832)]} 0 11
[junit4:junit4]   2> 10622 T216 C4 P21225 /update {wt=javabin&version=2} {delete=[2 (-1417427041590968320)]} 0 1
[junit4:junit4]   2> 10624 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10002 (-1417427041594114048)]} 0 0
[junit4:junit4]   2> 10636 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10636 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10637 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[2 (-1417427041596211200)]} 0 11
[junit4:junit4]   2> 10637 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[10002 (-1417427041599356928)]} 0 8
[junit4:junit4]   2> 10645 T217 C4 P21225 /update {wt=javabin&version=2} {add=[4 (1417427041611939840)]} 0 4
[junit4:junit4]   2> 10648 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10004 (1417427041612988416)]} 0 6
[junit4:junit4]   2> 10660 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10662 T232 C3 P21230 /update {wt=javabin&version=2} {add=[4 (1417427041621377024)]} 0 12
[junit4:junit4]   2> 10663 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10664 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10004 (1417427041623474176)]} 0 12
[junit4:junit4]   2> 10669 T218 C4 P21225 /update {wt=javabin&version=2} {add=[5 (1417427041638154240)]} 0 3
[junit4:junit4]   2> 10685 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10686 T234 C3 P21230 /update {wt=javabin&version=2} {add=[5 (1417427041646542848)]} 0 12
[junit4:junit4]   2> 10698 T216 C4 P21225 /update {wt=javabin&version=2} {add=[6 (1417427041664368640)]} 0 8
[junit4:junit4]   2> 10709 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10005 (1417427041640251392)]} 0 41
[junit4:junit4]   2> 10731 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=6 
[junit4:junit4]   2> 10732 T232 C3 P21230 /update {wt=javabin&version=2} {add=[6 (1417427041675902976)]} 0 30
[junit4:junit4]   2> 10739 T216 C4 P21225 /update {wt=javabin&version=2} {add=[7 (1417427041711554560)]} 0 3
[junit4:junit4]   2> 10742 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=20 
[junit4:junit4]   2> 10743 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10005 (1417427041687437312)]} 0 30
[junit4:junit4]   2> 10750 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10006 (1417427041723088896)]} 0 3
[junit4:junit4]   2> 10752 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10754 T232 C3 P21230 /update {wt=javabin&version=2} {add=[7 (1417427041718894592)]} 0 11
[junit4:junit4]   2> 10758 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[3 (-1417427041734623232)]} 0 0
[junit4:junit4]   2> 10765 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10766 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10006 (1417427041731477504)]} 0 11
[junit4:junit4]   2> 10766 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10767 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[3 (-1417427041738817536)]} 0 5
[junit4:junit4]   2> 10773 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10007 (1417427041747206144)]} 0 3
[junit4:junit4]   2> 10775 T218 C4 P21225 /update {wt=javabin&version=2} {add=[8 (1417427041749303296)]} 0 3
[junit4:junit4]   2> 10788 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10789 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10007 (1417427041754546176)]} 0 12
[junit4:junit4]   2> 10790 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10791 T234 C3 P21230 /update {wt=javabin&version=2} {add=[8 (1417427041756643328)]} 0 12
[junit4:junit4]   2> 10793 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[10003 (-1417427041771323392)]} 0 0
[junit4:junit4]   2> 10798 T216 C4 P21225 /update {wt=javabin&version=2} {add=[9 (1417427041773420544)]} 0 3
[junit4:junit4]   2> 10802 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10803 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[10003 (-1417427041775517696)]} 0 6
[junit4:junit4]   2> 10810 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10008 (1417427041786003456)]} 0 3
[junit4:junit4]   2> 10812 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10814 T232 C3 P21230 /update {wt=javabin&version=2} {add=[9 (1417427041780760576)]} 0 12
[junit4:junit4]   2> 10818 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[4 (-1417427041797537792)]} 0 0
[junit4:junit4]   2> 10823 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10824 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10008 (1417427041793343488)]} 0 10
[junit4:junit4]   2> 10825 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10827 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[4 (-1417427041802780672)]} 0 4
[junit4:junit4]   2> 10832 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10009 (1417427041809072128)]} 0 3
[junit4:junit4]   2> 10834 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10 (1417427041811169280)]} 0 3
[junit4:junit4]   2> 10846 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10847 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10009 (1417427041817460736)]} 0 10
[junit4:junit4]   2> 10849 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10850 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10 (1417427041819557888)]} 0 11
[junit4:junit4]   2> 10851 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[10004 (-1417427041832140800)]} 0 0
[junit4:junit4]   2> 10859 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10860 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[10004 (-1417427041837383680)]} 0 4
[junit4:junit4]   2> 10870 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10010 (1417427041845772288)]} 0 6
[junit4:junit4]   2> 10873 T216 C4 P21225 /update {wt=javabin&version=2} {add=[11 (1417427041835286528)]} 0 19
[junit4:junit4]   2> 10902 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=6 
[junit4:junit4]   2> 10903 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10010 (1417427041856258048)]} 0 29
[junit4:junit4]   2> 10905 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=20 
[junit4:junit4]   2> 10907 T234 C3 P21230 /update {wt=javabin&version=2} {add=[11 (1417427041859403776)]} 0 30
[junit4:junit4]   2> 10910 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10011 (1417427041890861056)]} 0 3
[junit4:junit4]   2> 10911 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[6 (-1417427041895055360)]} 0 0
[junit4:junit4]   2> 10919 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10920 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[6 (-1417427041900298240)]} 0 5
[junit4:junit4]   2> 10924 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10926 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10011 (1417427041899249664)]} 0 11
[junit4:junit4]   2> 10927 T216 C4 P21225 /update {wt=javabin&version=2} {add=[12 (1417427041908686848)]} 0 3
[junit4:junit4]   2> 10930 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[10006 (-1417427041914978304)]} 0 0
[junit4:junit4]   2> 10940 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10940 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10941 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[10006 (-1417427041920221184)]} 0 6
[junit4:junit4]   2> 10941 T234 C3 P21230 /update {wt=javabin&version=2} {add=[12 (1417427041916026880)]} 0 10
[junit4:junit4]   2> 10948 T218 C4 P21225 /update {wt=javabin&version=2} {add=[13 (1417427041930706944)]} 0 3
[junit4:junit4]   2> 10950 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10012 (1417427041930706945)]} 0 5
[junit4:junit4]   2> 10962 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10964 T233 C3 P21230 /update {wt=javabin&version=2} {add=[13 (1417427041938046976)]} 0 12
[junit4:junit4]   2> 10965 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10966 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10012 (1417427041939095552)]} 0 14
[junit4:junit4]   2> 10971 T217 C4 P21225 /update {wt=javabin&version=2} {add=[14 (1417427041954824192)]} 0 3
[junit4:junit4]   2> 10973 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10013 (1417427041956921344)]} 0 3
[junit4:junit4]   2> 10984 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10985 T233 C3 P21230 /update {wt=javabin&version=2} {add=[14 (1417427041962164224)]} 0 10
[junit4:junit4]   2> 10986 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10987 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10013 (1417427041964261376)]} 0 10
[junit4:junit4]   2> 10994 T217 C4 P21225 /update {wt=javabin&version=2} {add=[15 (1417427041978941440)]} 0 4
[junit4:junit4]   2> 10996 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10014 (1417427041978941441)]} 0 5
[junit4:junit4]   2> 11007 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11008 T234 C3 P21230 /update {wt=javabin&version=2} {add=[15 (1417427041986281472)]} 0 10
[junit4:junit4]   2> 11009 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11010 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10014 (1417427041988378624)]} 0 10
[junit4:junit4]   2> 11020 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10015 (1417427042003058688)]} 0 7
[junit4:junit4]   2> 11031 T218 C4 P21225 /update {wt=javabin&version=2} {add=[16 (1417427042002010112)]} 0 18
[junit4:junit4]   2> 11053 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=7 
[junit4:junit4]   2> 11054 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10015 (1417427042013544448)]} 0 30
[junit4:junit4]   2> 11061 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10016 (1417427042049196032)]} 0 3
[junit4:junit4]   2> 11063 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=18 
[junit4:junit4]   2> 11064 T234 C3 P21230 /update {wt=javabin&version=2} {add=[16 (1417427042026127360)]} 0 28
[junit4:junit4]   2> 11068 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[7 (-1417427042059681792)]} 0 2
[junit4:junit4]   2> 11075 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11077 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11077 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10016 (1417427042056536064)]} 0 12
[junit4:junit4]   2> 11078 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[7 (-1417427042063876096)]} 0 6
[junit4:junit4]   2> 11081 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10007 (-1417427042073313280)]} 0 0
[junit4:junit4]   2> 11085 T216 C4 P21225 /update {wt=javabin&version=2} {add=[17 (1417427042074361856)]} 0 3
[junit4:junit4]   2> 11090 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11092 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[10007 (-1417427042078556160)]} 0 6
[junit4:junit4]   2> 11098 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11099 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10017 (1417427042089041920)]} 0 3
[junit4:junit4]   2> 11100 T233 C3 P21230 /update {wt=javabin&version=2} {add=[17 (1417427042081701888)]} 0 11
[junit4:junit4]   2> 11104 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[9 (-1417427042097430528)]} 0 0
[junit4:junit4]   2> 11111 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11113 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[9 (-1417427042101624832)]} 0 5
[junit4:junit4]   2> 11113 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11114 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10017 (1417427042097430528)]} 0 10
[junit4:junit4]   2> 11120 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[10009 (-1417427042113159168)]} 0 1
[junit4:junit4]   2> 11121 T216 C4 P21225 /update {wt=javabin&version=2} {add=[18 (1417427042111062016)]} 0 4
[junit4:junit4]   2> 11127 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11129 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[10009 (-1417427042118402048)]} 0 5
[junit4:junit4]   2> 11136 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 11136 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10018 (1417427042127839232)]} 0 3
[junit4:junit4]   2> 11137 T234 C3 P21230 /update {wt=javabin&version=2} {add=[18 (1417427042119450624)]} 0 12
[junit4:junit4]   2> 11145 T217 C4 P21225 /update {wt=javabin&version=2} {add=[19 (1417427042137276416)]} 0 4
[junit4:junit4]   2> 11152 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11153 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10018 (1417427042137276416)]} 0 12
[junit4:junit4]   2> 11159 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11160 T234 C3 P21230 /update {wt=javabin&version=2} {add=[19 (1417427042145665024)]} 0 10
[junit4:junit4]   2> 11161 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10019 (1417427042154053632)]} 0 2
[junit4:junit4]   2> 11167 T218 C4 P21225 /update {wt=javabin&version=2} {add=[20 (1417427042160345088)]} 0 3
[junit4:junit4]   2> 11174 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11176 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10019 (1417427042161393664)]} 0 11
[junit4:junit4]   2> 11180 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11181 T233 C3 P21230 /update {wt=javabin&version=2} {add=[20 (1417427042167685120)]} 0 10
[junit4:junit4]   2> 11192 T216 C4 P21225 /update {wt=javabin&version=2} {add=[21 (1417427042183413760)]} 0 7
[junit4:junit4]   2> 11201 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10020 (1417427042177122304)]} 0 21
[junit4:junit4]   2> 11226 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 11227 T232 C3 P21230 /update {wt=javabin&version=2} {add=[21 (1417427042194948096)]} 0 30
[junit4:junit4]   2> 11234 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=17 
[junit4:junit4]   2> 11235 T218 C4 P21225 /update {wt=javabin&version=2} {add=[22 (1417427042231648256)]} 0 3
[junit4:junit4]   2> 11235 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10020 (1417427042205433856)]} 0 29
[junit4:junit4]   2> 11243 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10021 (1417427042240036864)]} 0 3
[junit4:junit4]   2> 11251 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 11252 T234 C3 P21230 /update {wt=javabin&version=2} {add=[22 (1417427042238988288)]} 0 13
[junit4:junit4]   2> 11256 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[15 (-1417427042256814080)]} 0 0
[junit4:junit4]   2> 11259 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11260 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10021 (1417427042247376896)]} 0 13
[junit4:junit4]   2> 11265 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11266 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[15 (-1417427042261008384)]} 0 6
[junit4:junit4]   2> 11267 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10022 (1417427042265202688)]} 0 3
[junit4:junit4]   2> 11273 T216 C4 P21225 /update {wt=javabin&version=2} {add=[23 (1417427042271494144)]} 0 4
[junit4:junit4]   2> 11280 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11281 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10022 (1417427042273591296)]} 0 9
[junit4:junit4]   2> 11285 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[10015 (-1417427042287222784)]} 0 0
[junit4:junit4]   2> 11286 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11291 T232 C3 P21230 /update {wt=javabin&version=2} {add=[23 (1417427042278834176)]} 0 14
[junit4:junit4]   2> 11296 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11297 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[18 (-1417427042298757120)]} 0 1
[junit4:junit4]   2> 11297 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[10015 (-1417427042294562816)]} 0 6
[junit4:junit4]   2> 11304 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10023 (1417427042304000000)]} 0 3
[junit4:junit4]   2> 11304 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11305 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[18 (-1417427042304000000)]} 0 4
[junit4:junit4]   2> 11311 T217 C4 P21225 /update {wt=javabin&version=2} {add=[24 (1417427042312388608)]} 0 3
[junit4:junit4]   2> 11316 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11317 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10023 (1417427042311340032)]} 0 9
[junit4:junit4]   2> 11321 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10018 (-1417427042324971520)]} 0 0
[junit4:junit4]   2> 11324 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11325 T233 C3 P21230 /update {wt=javabin&version=2} {add=[24 (1417427042319728640)]} 0 9
[junit4:junit4]   2> 11329 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11329 T216 C4 P21225 /update {wt=javabin&version=2} {delete=[21 (-1417427042333360128)]} 0 0
[junit4:junit4]   2> 11330 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[10018 (-1417427042329165824)]} 0 5
[junit4:junit4]   2> 11336 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10024 (1417427042338603008)]} 0 3
[junit4:junit4]   2> 11337 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11338 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[21 (-1417427042337554432)]} 0 5
[junit4:junit4]   2> 11344 T218 C4 P21225 /update {wt=javabin&version=2} {add=[25 (1417427042346991616)]} 0 3
[junit4:junit4]   2> 11349 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11350 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10024 (1417427042345943040)]} 0 10
[junit4:junit4]   2> 11354 T216 C4 P21225 /update {wt=javabin&version=2} {delete=[10021 (-1417427042359574528)]} 0 1
[junit4:junit4]   2> 11356 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11357 T234 C3 P21230 /update {wt=javabin&version=2} {add=[25 (1417427042353283072)]} 0 10
[junit4:junit4]   2> 11361 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11362 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[10021 (-1417427042363768832)]} 0 4
[junit4:junit4]   2> 11370 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10025 (1417427042372157440)]} 0 4
[junit4:junit4]   2> 11378 T217 C4 P21225 /update {wt=javabin&version=2} {add=[26 (1417427042366914560)]} 0 17
[junit4:junit4]   2> 11392 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11393 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10025 (1417427042381594624)]} 0 18
[junit4:junit4]   2> 11399 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=10 
[junit4:junit4]   2> 11399 T234 C3 P21230 /update {wt=javabin&version=2} {add=[26 (1417427042388934656)]} 0 17
[junit4:junit4]   2> 11400 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10026 (1417427042405711872)]} 0 2
[junit4:junit4]   2> 11403 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[22 (-1417427042410954752)]} 0 0
[junit4:junit4]   2> 11411 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11412 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[22 (-1417427042416197632)]} 0 4
[junit4:junit4]   2> 11412 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11413 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10026 (1417427042413051904)]} 0 9
[junit4:junit4]   2> 11417 T216 C4 P21225 /update {wt=javabin&version=2} {delete=[10022 (-1417427042425634816)]} 0 1
[junit4:junit4]   2> 11418 T218 C4 P21225 /update {wt=javabin&version=2} {add=[27 (1417427042424586240)]} 0 2
[junit4:junit4]   2> 11425 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11426 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[10022 (-1417427042430877696)]} 0 4
[junit4:junit4]   2> 11430 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11431 T232 C3 P21230 /update {wt=javabin&version=2} {add=[27 (1417427042431926272)]} 0 8
[junit4:junit4]   2> 11432 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10027 (1417427042439266304)]} 0 2
[junit4:junit4]   2> 11435 T216 C4 P21225 /update {wt=javabin&version=2} {delete=[26 (-1417427042444509184)]} 0 0
[junit4:junit4]   2> 11442 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11443 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[26 (-1417427042447654912)]} 0 5
[junit4:junit4]   2> 11443 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11444 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10027 (1417427042446606336)]} 0 8
[junit4:junit4]   2> 11448 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[10026 (-1417427042458140672)]} 0 0
[junit4:junit4]   2> 11450 T218 C4 P21225 /update {wt=javabin&version=2} {add=[28 (1417427042457092096)]} 0 3
[junit4:junit4]   2> 11455 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11456 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[10026 (-1417427042462334976)]} 0 5
[junit4:junit4]   2> 11460 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11461 T234 C3 P21230 /update {wt=javabin&version=2} {add=[28 (1417427042464432128)]} 0 8
[junit4:junit4]   2> 11462 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10028 (1417427042470723584)]} 0 2
[junit4:junit4]   2> 11465 T217 C4 P21225 /update {wt=javabin&version=2} {delete=[27 (-1417427042475966464)]} 0 0
[junit4:junit4]   2> 11472 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11473 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11473 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10028 (1417427042477015040)]} 0 8
[junit4:junit4]   2> 11474 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[27 (-1417427042481209344)]} 0 5
[junit4:junit4]   2> 11477 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10027 (-1417427042488549376)]} 0 0
[junit4:junit4]   2> 11479 T216 C4 P21225 /update {wt=javabin&version=2} {add=[29 (1417427042489597952)]} 0 2
[junit4:junit4]   2> 11484 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11485 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[10027 (-1417427042492743680)]} 0 5
[junit4:junit4]   2> 11491 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11491 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10029 (1417427042501132288)]} 0 2
[junit4:junit4]   2> 11493 T233 C3 P21230 /update {wt=javabin&version=2} {add=[29 (1417427042494840832)]} 0 11
[junit4:junit4]   2> 11496 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[29 (-1417427042508472320)]} 0 0
[junit4:junit4]   2> 11503 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11504 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11505 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10029 (1417427042508472320)]} 0 9
[junit4:junit4]   2> 11505 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[29 (-1417427042513715200)]} 0 5
[junit4:junit4]   2> 11509 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10029 (-1417427042522103808)]} 0 0
[junit4:junit4]   2> 11511 T217 C4 P21225 /update {wt=javabin&version=2} {add=[30 (1417427042522103809)]} 0 2
[junit4:junit4]   2> 11516 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11517 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[10029 (-1417427042526298112)]} 0 4
[junit4:junit4]   2> 11522 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11523 T232 C3 P21230 /update {wt=javabin&version=2} {add=[30 (1417427042529443840)]} 0 7
[junit4:junit4]   2> 11532 T218 C4 P21225 /update {wt=javabin&version=2} {add=[31 (1417427042542026752)]} 0 4
[junit4:junit4]   2> 11538 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10030 (1417427042534686720)]} 0 17
[junit4:junit4]   2> 11562 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=12 
[junit4:junit4]   2> 11562 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 11563 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10030 (1417427042556706816)]} 0 21
[junit4:junit4]   2> 11563 T234 C3 P21230 /update {wt=javabin&version=2} {add=[31 (1417427042551463936)]} 0 26
[junit4:junit4]   2> 11568 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[31 (-1417427042583969792)]} 0 1
[junit4:junit4]   2> 11569 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10031 (1417427042582921216)]} 0 2
[junit4:junit4]   2> 11576 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11577 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[31 (-1417427042588164096)]} 0 6
[junit4:junit4]   2> 11581 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11582 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10031 (1417427042590261248)]} 0 9
[junit4:junit4]   2> 11583 T216 C4 P21225 /update {wt=javabin&version=2} {add=[32 (1417427042597601280)]} 0 3
[junit4:junit4]   2> 11586 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10031 (-1417427042602844160)]} 0 1
[junit4:junit4]   2> 11593 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11594 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[10031 (-1417427042607038464)]} 0 4
[junit4:junit4]   2> 11595 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11596 T234 C3 P21230 /update {wt=javabin&version=2} {add=[32 (1417427042603892736)]} 0 10
[junit4:junit4]   2> 11600 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10032 (1417427042615427072)]} 0 2
[junit4:junit4]   2> 11602 T216 C4 P21225 /update {wt=javabin&version=2} {add=[33 (1417427042617524224)]} 0 3
[junit4:junit4]   2> 11611 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11612 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10032 (1417427042621718528)]} 0 9
[junit4:junit4]   2> 11613 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11614 T233 C3 P21230 /update {wt=javabin&version=2} {add=[33 (1417427042623815680)]} 0 9
[junit4:junit4]   2> 11617 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10033 (1417427042634301440)]} 0 2
[junit4:junit4]   2> 11620 T217 C4 P21225 /update {wt=javabin&version=2} {add=[34 (1417427042636398592)]} 0 3
[junit4:junit4]   2> 11628 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11629 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10033 (1417427042640592896)]} 0 8
[junit4:junit4]   2> 11632 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11633 T232 C3 P21230 /update {wt=javabin&version=2} {add=[34 (1417427042642690048)]} 0 9
[junit4:junit4]   2> 11635 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10034 (1417427042652127232)]} 0 2
[junit4:junit4]   2> 11637 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[34 (-1417427042656321536)]} 0 1
[junit4:junit4]   2> 11644 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11646 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11646 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[34 (-1417427042660515840)]} 0 5
[junit4:junit4]   2> 11647 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10034 (1417427042658418688)]} 0 9
[junit4:junit4]   2> 11652 T216 C4 P21225 /update {wt=javabin&version=2} {delete=[10034 (-1417427042672050176)]} 0 0
[junit4:junit4]   2> 11653 T217 C4 P21225 /update {wt=javabin&version=2} {add=[35 (1417427042671001600)]} 0 3
[junit4:junit4]   2> 11660 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11661 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[10034 (-1417427042676244480)]} 0 5
[junit4:junit4]   2> 11664 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11665 T234 C3 P21230 /update {wt=javabin&version=2} {add=[35 (1417427042677293056)]} 0 8
[junit4:junit4]   2> 11673 T216 C4 P21225 /update {wt=javabin&version=2} {add=[36 (1417427042689875968)]} 0 4
[junit4:junit4]   2> 11676 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10035 (1417427042685681664)]} 0 11
[junit4:junit4]   2> 11701 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11702 T233 C3 P21230 /update {wt=javabin&version=2} {add=[36 (1417427042699313152)]} 0 24
[junit4:junit4]   2> 11705 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=16 
[junit4:junit4]   2> 11707 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10035 (1417427042703507456)]} 0 26
[junit4:junit4]   2> 11710 T217 C4 P21225 /update {wt=javabin&version=2} {add=[37 (1417427042729721856)]} 0 3
[junit4:junit4]   2> 11713 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10036 (1417427042733916160)]} 0 2
[junit4:junit4]   2> 11721 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11722 T234 C3 P21230 /update {wt=javabin&version=2} {add=[37 (1417427042737061888)]} 0 8
[junit4:junit4]   2> 11724 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11726 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10036 (1417427042740207616)]} 0 9
[junit4:junit4]   2> 11726 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[36 (-1417427042749644800)]} 0 1
[junit4:junit4]   2> 11732 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10037 (1417427042753839104)]} 0 2
[junit4:junit4]   2> 11734 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11735 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[36 (-1417427042753839104)]} 0 5
[junit4:junit4]   2> 11741 T216 C4 P21225 /update {wt=javabin&version=2} {add=[38 (1417427042763276288)]} 0 2
[junit4:junit4]   2> 11741 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11743 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10037 (1417427042760130560)]} 0 8
[junit4:junit4]   2> 11746 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10036 (-1417427042770616320)]} 0 1
[junit4:junit4]   2> 11753 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 11754 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11754 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[10036 (-1417427042774810624)]} 0 4
[junit4:junit4]   2> 11755 T233 C3 P21230 /update {wt=javabin&version=2} {add=[38 (1417427042770616320)]} 0 9
[junit4:junit4]   2> 11761 T216 C4 P21225 /update {wt=javabin&version=2} {add=[39 (1417427042784247808)]} 0 2
[junit4:junit4]   2> 11762 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10038 (1417427042784247809)]} 0 3
[junit4:junit4]   2> 11771 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11772 T234 C3 P21230 /update {wt=javabin&version=2} {add=[39 (1417427042790539264)]} 0 7
[junit4:junit4]   2> 11773 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11774 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10038 (1417427042791587840)]} 0 9
[junit4:junit4]   2> 11776 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[39 (-1417427042802073600)]} 0 1
[junit4:junit4]   2> 11779 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10039 (1417427042804170752)]} 0 2
[junit4:junit4]   2> 11783 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11783 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[39 (-1417427042805219328)]} 0 4
[junit4:junit4]   2> 11788 T217 C4 P21225 /update {wt=javabin&version=2} {add=[40 (1417427042813607936)]} 0 2
[junit4:junit4]   2> 11788 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11789 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10039 (1417427042809413632)]} 0 7
[junit4:junit4]   2> 11793 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10039 (-1417427042818850816)]} 0 1
[junit4:junit4]   2> 11797 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11798 T233 C3 P21230 /update {wt=javabin&version=2} {add=[40 (1417427042818850816)]} 0 7
[junit4:junit4]   2> 11800 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 11803 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[10039 (-1417427042824093696)]} 0 7
[junit4:junit4]   2> 11811 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10040 (1417427042835628032)]} 0 4
[junit4:junit4]   2> 11959 T216 C4 P21225 /update {wt=javabin&version=2} {add=[41 (1417427042832482304)]} 0 155
[junit4:junit4]   2> 11973 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 11974 T232 C3 P21230 /update {wt=javabin&version=2} {add=[41 (1417427042998157312)]} 0 12
[junit4:junit4]   2> 11982 T217 C4 P21225 /update {wt=javabin&version=2} {add=[42 (1417427043014934528)]} 0 3
[junit4:junit4]   2> 11992 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 11993 T233 C3 P21230 /update {wt=javabin&version=2} {add=[42 (1417427043022274560)]} 0 8
[junit4:junit4]   2> 11997 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[42 (-1417427043033808896)]} 0 0
[junit4:junit4]   2> 12004 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 12005 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[42 (-1417427043038003200)]} 0 5
[junit4:junit4]   2> 12010 T216 C4 P21225 /update {wt=javabin&version=2} {add=[43 (1417427043046391808)]} 0 2
[junit4:junit4]   2> 12019 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12020 T233 C3 P21230 /update {wt=javabin&version=2} {add=[43 (1417427043051634688)]} 0 7
[junit4:junit4]   2> 12026 T217 C4 P21225 /update {wt=javabin&version=2} {add=[44 (1417427043062120448)]} 0 3
[junit4:junit4]   2> 12035 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12036 T232 C3 P21230 /update {wt=javabin&version=2} {add=[44 (1417427043068411904)]} 0 7
[junit4:junit4]   2> 12041 T218 C4 P21225 /update {wt=javabin&version=2} {add=[45 (1417427043078897664)]} 0 2
[junit4:junit4]   2> 12050 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12051 T233 C3 P21230 /update {wt=javabin&version=2} {add=[45 (1417427043084140544)]} 0 7
[junit4:junit4]   2> 12056 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=95 
[junit4:junit4]   2> 12056 T216 C4 P21225 /update {wt=javabin&version=2} {add=[46 (1417427043094626304)]} 0 2
[junit4:junit4]   2> 12057 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10040 (1417427042842968064)]} 0 243
[junit4:junit4]   2> 12062 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10041 (1417427043100917760)]} 0 2
[junit4:junit4]   2> 12065 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12066 T232 C3 P21230 /update {wt=javabin&version=2} {add=[46 (1417427043099869184)]} 0 7
[junit4:junit4]   2> 12070 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[45 (-1417427043110354944)]} 0 1
[junit4:junit4]   2> 12072 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12073 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10041 (1417427043106160640)]} 0 8
[junit4:junit4]   2> 12077 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 12078 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[45 (-1417427043114549248)]} 0 5
[junit4:junit4]   2> 12078 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10042 (1417427043117694976)]} 0 2
[junit4:junit4]   2> 12083 T217 C4 P21225 /update {wt=javabin&version=2} {add=[47 (1417427043122937856)]} 0 2
[junit4:junit4]   2> 12089 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 12090 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10042 (1417427043122937856)]} 0 8
[junit4:junit4]   2> 12093 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12094 T233 C3 P21230 /update {wt=javabin&version=2} {add=[47 (1417427043128180736)]} 0 8
[junit4:junit4]   2> 12094 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10042 (-1417427043135520768)]} 0 1
[junit4:junit4]   2> 12101 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 12102 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[10042 (-1417427043139715072)]} 0 5
[junit4:junit4]   2> 12109 T216 C4 P21225 /update {wt=javabin&version=2} {add=[48 (1417427043139715072)]} 0 12
[junit4:junit4]   2> 12111 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10043 (1417427043148103680)]} 0 5
[junit4:junit4]   2> 12132 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 12134 T232 C3 P21230 /update {wt=javabin&version=2} {add=[48 (1417427043156492288)]} 0 21
[junit4:junit4]   2> 12134 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=13 
[junit4:junit4]   2> 12135 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10043 (1417427043157540864)]} 0 20
[junit4:junit4]   2> 12139 T218 C4 P21225 /update {wt=javabin&version=2} {add=[49 (1417427043181658112)]} 0 2
[junit4:junit4]   2> 12141 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10044 (1417427043182706688)]} 0 2
[junit4:junit4]   2> 12151 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 12152 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 12152 T232 C3 P21230 /update {wt=javabin&version=2} {add=[10044 (1417427043187949568)]} 0 9
[junit4:junit4]   2> 12153 T234 C3 P21230 /update {wt=javabin&version=2} {add=[49 (1417427043187949569)]} 0 10
[junit4:junit4]   2> 12159 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[47 (-1417427043203678209)]} 0 0
[junit4:junit4]   2> 12160 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10045 (1417427043203678208)]} 0 2
[junit4:junit4]   2> 12207 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 12208 T233 C3 P21230 /update {wt=javabin&version=2} {delete=[47 (-1417427043249815552)]} 0 5
[junit4:junit4]   2> 12210 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12211 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10045 (1417427043250864128)]} 0 7
[junit4:junit4]   2> 12213 T216 C4 P21225 /update {wt=javabin&version=2} {add=[50 (1417427043259252736)]} 0 2
[junit4:junit4]   2> 12216 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10046 (1417427043262398464)]} 0 2
[junit4:junit4]   2> 12222 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12223 T232 C3 P21230 /update {wt=javabin&version=2} {add=[50 (1417427043263447040)]} 0 7
[junit4:junit4]   2> 12225 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12226 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10046 (1417427043267641344)]} 0 7
[junit4:junit4]   2> 12227 T216 C4 P21225 /update {wt=javabin&version=2} {delete=[50 (-1417427043273932800)]} 0 1
[junit4:junit4]   2> 12230 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[10045 (-1417427043277078528)]} 0 1
[junit4:junit4]   2> 12233 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 12233 T234 C3 P21230 /update {wt=javabin&version=2} {delete=[50 (-1417427043277078528)]} 0 4
[junit4:junit4]   2> 12236 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 12237 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[10045 (-1417427043280224256)]} 0 4
[junit4:junit4]   2> 12238 T217 C4 P21225 /update {wt=javabin&version=2} {add=[51 (1417427043284418560)]} 0 2
[junit4:junit4]   2> 12241 T216 C4 P21225 /update {wt=javabin&version=2} {add=[10047 (1417427043288612864)]} 0 2
[junit4:junit4]   2> 12247 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12248 T233 C3 P21230 /update {wt=javabin&version=2} {add=[51 (1417427043289661440)]} 0 7
[junit4:junit4]   2> 12250 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12250 T234 C3 P21230 /update {wt=javabin&version=2} {add=[10047 (1417427043292807168)]} 0 6
[junit4:junit4]   2> 12252 T218 C4 P21225 /update {wt=javabin&version=2} {delete=[51 (-1417427043300147200)]} 0 1
[junit4:junit4]   2> 12255 T217 C4 P21225 /update {wt=javabin&version=2} {add=[10048 (1417427043302244352)]} 0 2
[junit4:junit4]   2> 12259 T248 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 12260 T232 C3 P21230 /update {wt=javabin&version=2} {delete=[51 (-1417427043304341504)]} 0 5
[junit4:junit4]   2> 12265 T249 C2 P21235 REQ /update {distrib.from=http://127.0.0.1:21230/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 12266 T233 C3 P21230 /update {wt=javabin&version=2} {add=[10048 (1417427043308535808)]} 0 8
[junit4:junit4]   2> 12272 T218 C4 P21225 /update {wt=javabin&version=2} {add=[10049 (1417427043320070144)]} 0 3
[junit4:ju

[...truncated too long message...]

.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    7) Thread[id=339, name=qtp551926596-339 Acceptor3 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    8) Thread[id=404, name=qtp680254087-404 Acceptor0 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([10754BD6AFFFB7CB]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=407, name=qtp680254087-407 Acceptor3 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    2) Thread[id=406, name=qtp680254087-406 Acceptor2 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    3) Thread[id=405, name=qtp680254087-405 Acceptor1 SelectChannelConnector@0.0.0.0:22157, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    4) Thread[id=338, name=qtp551926596-338 Acceptor2 SelectChannelConnector@0.0.0.0:57705, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    5) Thread[id=336, name=qtp551926596-336 Acceptor0 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    6) Thread[id=337, name=qtp551926596-337 Acceptor1 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    7) Thread[id=339, name=qtp551926596-339 Acceptor3 SelectChannelConnector@0.0.0.0:57705, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    8) Thread[id=404, name=qtp680254087-404 Acceptor0 SelectChannelConnector@0.0.0.0:22157, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([10754BD6AFFFB7CB]:0)
[junit4:junit4] Completed on J0 in 710.58s, 1 test, 2 errors <<< FAILURES!

[...truncated 89 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 757 ignored

Total time: 32 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message