lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-java7 - Build # 607 - Still Failing
Date Fri, 02 Nov 2012 12:24:23 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-java7/607/

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=83, name=qtp1635915482-83 Acceptor3 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=80, name=qtp1635915482-80 Acceptor0 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=82, name=qtp1635915482-82 Acceptor2 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=32, name=qtp1013516039-32 Acceptor2 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=33, name=qtp1013516039-33 Acceptor3 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=101, name=qtp370838838-101 Acceptor3 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=100, name=qtp370838838-100 Acceptor2 SelectChannelConnector@0.0.0.0:41359, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=81, name=qtp1635915482-81 Acceptor1 SelectChannelConnector@0.0.0.0:48884, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=99, name=qtp370838838-99 Acceptor1 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=98, name=qtp370838838-98 Acceptor0 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=31, name=qtp1013516039-31 Acceptor1 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=30, name=qtp1013516039-30 Acceptor0 SelectChannelConnector@0.0.0.0:44571, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=83, name=qtp1635915482-83 Acceptor3 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=80, name=qtp1635915482-80 Acceptor0 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=82, name=qtp1635915482-82 Acceptor2 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=32, name=qtp1013516039-32 Acceptor2 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=33, name=qtp1013516039-33 Acceptor3 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=101, name=qtp370838838-101 Acceptor3 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=100, name=qtp370838838-100 Acceptor2 SelectChannelConnector@0.0.0.0:41359, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=81, name=qtp1635915482-81 Acceptor1 SelectChannelConnector@0.0.0.0:48884, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=99, name=qtp370838838-99 Acceptor1 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=98, name=qtp370838838-98 Acceptor0 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=31, name=qtp1013516039-31 Acceptor1 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=30, name=qtp1013516039-30 Acceptor0 SelectChannelConnector@0.0.0.0:44571, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([97D4AB69C437B5E3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=100, name=qtp370838838-100 Acceptor2 SelectChannelConnector@0.0.0.0:41359, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=83, name=qtp1635915482-83 Acceptor3 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=80, name=qtp1635915482-80 Acceptor0 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=81, name=qtp1635915482-81 Acceptor1 SelectChannelConnector@0.0.0.0:48884, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=82, name=qtp1635915482-82 Acceptor2 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=99, name=qtp370838838-99 Acceptor1 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=98, name=qtp370838838-98 Acceptor0 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=31, name=qtp1013516039-31 Acceptor1 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=30, name=qtp1013516039-30 Acceptor0 SelectChannelConnector@0.0.0.0:44571, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=32, name=qtp1013516039-32 Acceptor2 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=33, name=qtp1013516039-33 Acceptor3 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=101, name=qtp370838838-101 Acceptor3 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=100, name=qtp370838838-100 Acceptor2 SelectChannelConnector@0.0.0.0:41359, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=83, name=qtp1635915482-83 Acceptor3 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=80, name=qtp1635915482-80 Acceptor0 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=81, name=qtp1635915482-81 Acceptor1 SelectChannelConnector@0.0.0.0:48884, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=82, name=qtp1635915482-82 Acceptor2 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=99, name=qtp370838838-99 Acceptor1 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=98, name=qtp370838838-98 Acceptor0 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=31, name=qtp1013516039-31 Acceptor1 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=30, name=qtp1013516039-30 Acceptor0 SelectChannelConnector@0.0.0.0:44571, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=32, name=qtp1013516039-32 Acceptor2 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=33, name=qtp1013516039-33 Acceptor3 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=101, name=qtp370838838-101 Acceptor3 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([97D4AB69C437B5E3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=217, name=qtp118471033-217 Acceptor3 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=282, name=qtp821896153-282 Acceptor0 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=199, name=qtp574732811-199 Acceptor3 SelectChannelConnector@0.0.0.0:19836, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=285, name=qtp821896153-285 Acceptor3 SelectChannelConnector@0.0.0.0:48463, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=198, name=qtp574732811-198 Acceptor2 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=216, name=qtp118471033-216 Acceptor2 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=196, name=qtp574732811-196 Acceptor0 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=215, name=qtp118471033-215 Acceptor1 SelectChannelConnector@0.0.0.0:44503, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=214, name=qtp118471033-214 Acceptor0 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=283, name=qtp821896153-283 Acceptor1 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=197, name=qtp574732811-197 Acceptor1 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=284, name=qtp821896153-284 Acceptor2 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest: 
   1) Thread[id=217, name=qtp118471033-217 Acceptor3 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=282, name=qtp821896153-282 Acceptor0 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=199, name=qtp574732811-199 Acceptor3 SelectChannelConnector@0.0.0.0:19836, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=285, name=qtp821896153-285 Acceptor3 SelectChannelConnector@0.0.0.0:48463, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=198, name=qtp574732811-198 Acceptor2 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=216, name=qtp118471033-216 Acceptor2 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=196, name=qtp574732811-196 Acceptor0 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=215, name=qtp118471033-215 Acceptor1 SelectChannelConnector@0.0.0.0:44503, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=214, name=qtp118471033-214 Acceptor0 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=283, name=qtp821896153-283 Acceptor1 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=197, name=qtp574732811-197 Acceptor1 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=284, name=qtp821896153-284 Acceptor2 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([97D4AB69C437B5E3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=216, name=qtp118471033-216 Acceptor2 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=196, name=qtp574732811-196 Acceptor0 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=215, name=qtp118471033-215 Acceptor1 SelectChannelConnector@0.0.0.0:44503, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=217, name=qtp118471033-217 Acceptor3 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=282, name=qtp821896153-282 Acceptor0 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=214, name=qtp118471033-214 Acceptor0 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=283, name=qtp821896153-283 Acceptor1 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=199, name=qtp574732811-199 Acceptor3 SelectChannelConnector@0.0.0.0:19836, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=197, name=qtp574732811-197 Acceptor1 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=284, name=qtp821896153-284 Acceptor2 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=285, name=qtp821896153-285 Acceptor3 SelectChannelConnector@0.0.0.0:48463, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=198, name=qtp574732811-198 Acceptor2 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=216, name=qtp118471033-216 Acceptor2 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=196, name=qtp574732811-196 Acceptor0 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=215, name=qtp118471033-215 Acceptor1 SelectChannelConnector@0.0.0.0:44503, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=217, name=qtp118471033-217 Acceptor3 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=282, name=qtp821896153-282 Acceptor0 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=214, name=qtp118471033-214 Acceptor0 SelectChannelConnector@0.0.0.0:44503, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=283, name=qtp821896153-283 Acceptor1 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=199, name=qtp574732811-199 Acceptor3 SelectChannelConnector@0.0.0.0:19836, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=197, name=qtp574732811-197 Acceptor1 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=284, name=qtp821896153-284 Acceptor2 SelectChannelConnector@0.0.0.0:48463, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=285, name=qtp821896153-285 Acceptor3 SelectChannelConnector@0.0.0.0:48463, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=198, name=qtp574732811-198 Acceptor2 SelectChannelConnector@0.0.0.0:19836, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([97D4AB69C437B5E3]:0)


FAILED:  org.apache.solr.cloud.RecoveryZkTest.testDistribSearch

Error Message:
Server at http://127.0.0.1:48463/solr/collection1 returned non ok status:404, message:Can not find: /solr/collection1/select

Stack Trace:
org.apache.solr.common.SolrException: Server at http://127.0.0.1:48463/solr/collection1 returned non ok status:404, message:Can not find: /solr/collection1/select
	at __randomizedtesting.SeedInfo.seed([97D4AB69C437B5E3:16322571B368D5DF]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:92)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 8340 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4]   2> 3 T180 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-RecoveryZkTest-1351858862598
[junit4:junit4]   2> 4 T180 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 8 T181 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 8 T181 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 9 T181 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 106 T180 oasc.ZkTestServer.run start zk server on port:20217
[junit4:junit4]   2> 106 T180 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:20217 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@771c674a
[junit4:junit4]   2> 107 T186 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:20217
[junit4:junit4]   2> 107 T180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 108 T186 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:20217, initiating session
[junit4:junit4]   2> 109 T182 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:59869
[junit4:junit4]   2> 109 T182 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:59869
[junit4:junit4]   2> 110 T184 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 120 T184 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1113a0c0000 with negotiated timeout 10000 for client /140.211.11.196:59869
[junit4:junit4]   2> 120 T186 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:20217, sessionid = 0x13ac1113a0c0000, negotiated timeout = 10000
[junit4:junit4]   2> 121 T187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@771c674a name:ZooKeeperConnection Watcher:127.0.0.1:20217 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 121 T180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 122 T180 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 132 T185 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1113a0c0000
[junit4:junit4]   2> 142 T187 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 142 T180 oaz.ZooKeeper.close Session: 0x13ac1113a0c0000 closed
[junit4:junit4]   2> 143 T182 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:59869 which had sessionid 0x13ac1113a0c0000
[junit4:junit4]   2> 143 T180 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:20217/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@69dd174a
[junit4:junit4]   2> 144 T188 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:20217
[junit4:junit4]   2> 144 T180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 145 T188 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:20217, initiating session
[junit4:junit4]   2> 145 T182 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:49570
[junit4:junit4]   2> 145 T182 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:49570
[junit4:junit4]   2> 147 T184 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1113a0c0001 with negotiated timeout 10000 for client /140.211.11.196:49570
[junit4:junit4]   2> 147 T188 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:20217, sessionid = 0x13ac1113a0c0001, negotiated timeout = 10000
[junit4:junit4]   2> 148 T189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69dd174a name:ZooKeeperConnection Watcher:127.0.0.1:20217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 148 T180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 148 T180 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 158 T180 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 161 T180 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 163 T180 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 167 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 168 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 198 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 199 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 208 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 209 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 221 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 222 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 228 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 229 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 236 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 237 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 247 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 248 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 257 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 258 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 265 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 265 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 272 T180 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 273 T180 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 276 T185 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1113a0c0001
[junit4:junit4]   2> 282 T189 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 282 T180 oaz.ZooKeeper.close Session: 0x13ac1113a0c0001 closed
[junit4:junit4]   2> 283 T182 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:49570 which had sessionid 0x13ac1113a0c0001
[junit4:junit4]   2> 864 T180 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1134 T180 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:19836
[junit4:junit4]   2> 1165 T180 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1166 T180 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1166 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1198 T180 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1199 T180 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1199 T180 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1199 T180 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1200 T180 oasc.CoreContainer.<init> New CoreContainer 402288927
[junit4:junit4]   2> 1200 T180 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1201 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1241 T180 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1254 T180 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:20217/solr
[junit4:junit4]   2> 1258 T180 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:20217/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@5add9fcc
[junit4:junit4]   2> 1259 T200 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:20217
[junit4:junit4]   2> 1259 T180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1261 T200 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:20217, initiating session
[junit4:junit4]   2> 1261 T182 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:28076
[junit4:junit4]   2> 1261 T182 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:28076
[junit4:junit4]   2> 1266 T184 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1113a0c0002 with negotiated timeout 5000 for client /140.211.11.196:28076
[junit4:junit4]   2> 1266 T200 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:20217, sessionid = 0x13ac1113a0c0002, negotiated timeout = 5000
[junit4:junit4]   2> 1267 T201 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5add9fcc name:ZooKeeperConnection Watcher:127.0.0.1:20217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1267 T180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1286 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1304 T180 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1306 T180 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:19836_solr
[junit4:junit4]   2> 1307 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:19836_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:19836_solr
[junit4:junit4]   2> 1308 T180 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:19836_solr
[junit4:junit4]   2> 1311 T180 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1311 T180 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1311 T180 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1312 T180 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1312 T180 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1312 T180 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1313 T180 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1313 T180 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1313 T180 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1314 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1327 T180 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1335 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1336 T180 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1338 T180 oasc.Overseer.start Overseer (id=88595422419615746-127.0.0.1:19836_solr-n_0000000000) starting
[junit4:junit4]   2> 1339 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1354 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1355 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1360 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1362 T203 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1362 T180 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1365 T180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1382 T202 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1427 T180 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1427 T180 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 1428 T180 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1429 T180 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 1430 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1431 T180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1432 T180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1502 T180 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1557 T180 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1559 T180 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1566 T180 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2013 T180 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2022 T180 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2025 T180 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2044 T180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2049 T180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2052 T180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2054 T180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2056 T180 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351858862598/control/data/
[junit4:junit4]   2> 2056 T180 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13d380dc
[junit4:junit4]   2> 2078 T180 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351858862598/control/data/index/
[junit4:junit4]   2> 2079 T180 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351858862598/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2080 T180 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351858862598/control/data/index forceNew:false
[junit4:junit4]   2> 2161 T180 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351858862598/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2161 T180 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2162 T180 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2164 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2164 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2165 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2165 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2166 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2166 T180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2166 T180 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2167 T180 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2167 T180 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2168 T180 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2168 T180 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2169 T180 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2169 T180 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2170 T180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2170 T180 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2171 T180 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2171 T180 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2172 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2172 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2173 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2173 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2174 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2174 T180 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2175 T180 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2175 T180 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2183 T180 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2203 T180 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2204 T180 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 2208 T180 oass.SolrIndexSearcher.<init> Opening Searcher@396c4f69 main
[junit4:junit4]   2> 2209 T180 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2209 T180 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2209 T180 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2299 T180 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2363 T208 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@396c4f69 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2370 T180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2372 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x53 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2384 T180 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2384 T180 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:19836/solr shardId:control_shard
[junit4:junit4]   2> 2388 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2389 T202 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:19836_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:19836/solr"}
[junit4:junit4]   2> 2404 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 2407 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x5b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2413 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2424 T180 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 2431 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:delete cxid:0x74 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2432 T180 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2433 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x75 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2435 T180 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2436 T180 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2441 T180 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:19836/solr/collection1/
[junit4:junit4]   2> 2442 T180 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2442 T180 oasc.SyncStrategy.syncToMe http://127.0.0.1:19836/solr/collection1/ has no replicas
[junit4:junit4]   2> 2443 T180 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:19836/solr/collection1/
[junit4:junit4]   2> 2443 T180 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 2509 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0002 type:create cxid:0x7f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2918 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2940 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2984 T180 oasc.ZkController.register We are http://127.0.0.1:19836/solr/collection1/ and leader is http://127.0.0.1:19836/solr/collection1/
[junit4:junit4]   2> 2984 T180 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:19836/solr
[junit4:junit4]   2> 2985 T180 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2985 T180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2988 T180 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2989 T180 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2989 T180 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3072 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3273 T180 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3276 T180 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:44503
[junit4:junit4]   2> 3277 T180 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3278 T180 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695
[junit4:junit4]   2> 3278 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695/'
[junit4:junit4]   2> 3331 T180 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3331 T180 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3332 T180 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695
[junit4:junit4]   2> 3332 T180 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695/solr.xml
[junit4:junit4]   2> 3333 T180 oasc.CoreContainer.<init> New CoreContainer 1145621097
[junit4:junit4]   2> 3333 T180 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695/'
[junit4:junit4]   2> 3334 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695/'
[junit4:junit4]   2> 3369 T180 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3386 T180 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:20217/solr
[junit4:junit4]   2> 3387 T180 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:20217/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@48ed4623
[junit4:junit4]   2> 3388 T218 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:20217
[junit4:junit4]   2> 3388 T180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3390 T218 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:20217, initiating session
[junit4:junit4]   2> 3390 T182 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:22292
[junit4:junit4]   2> 3390 T182 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:22292
[junit4:junit4]   2> 3393 T184 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1113a0c0003 with negotiated timeout 5000 for client /140.211.11.196:22292
[junit4:junit4]   2> 3393 T218 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:20217, sessionid = 0x13ac1113a0c0003, negotiated timeout = 5000
[junit4:junit4]   2> 3393 T219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48ed4623 name:ZooKeeperConnection Watcher:127.0.0.1:20217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3393 T180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3394 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3396 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3407 T180 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44503_solr
[junit4:junit4]   2> 3407 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:44503_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:44503_solr
[junit4:junit4]   2> 3408 T180 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44503_solr
[junit4:junit4]   2> 3418 T201 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 3418 T180 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3419 T180 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3419 T180 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3420 T180 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3420 T180 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3420 T180 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3421 T180 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3421 T180 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3422 T180 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3422 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3447 T180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3452 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3453 T202 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:19836_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:19836/solr"}
[junit4:junit4]   2> 3453 T180 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695/collection1
[junit4:junit4]   2> 3454 T180 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 3473 T180 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 3473 T180 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 3476 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695/collection1/'
[junit4:junit4]   2> 3476 T219 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 3476 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 3477 T180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty1-1351858865695/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 3478 T180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty1-1351858865695/collection1/lib/README' to classloader
[junit4:junit4]   2> 3538 T180 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 3614 T180 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 3616 T180 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 3626 T180 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4382 T180 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4391 T180 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4394 T180 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4413 T180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4418 T180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4422 T180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4424 T180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4425 T180 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351858865695/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty1/
[junit4:junit4]   2> 4426 T180 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13d380dc
[junit4:junit4]   2> 4427 T180 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty1/index/
[junit4:junit4]   2> 4428 T180 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4428 T180 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty1/index forceNew:false
[junit4:junit4]   2> 4433 T180 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty1/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4433 T180 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4434 T180 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 4436 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 4436 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 4437 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 4437 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 4438 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 4438 T180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 4438 T180 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 4439 T180 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 4440 T180 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 4440 T180 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 4441 T180 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 4441 T180 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 4442 T180 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 4442 T180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 4442 T180 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 4443 T180 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 4444 T180 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4444 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4445 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4445 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4446 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4446 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4447 T180 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4447 T180 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 4448 T180 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 4463 T180 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 4468 T180 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 4469 T180 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 4473 T180 oass.SolrIndexSearcher.<init> Opening Searcher@6f5a6472 main
[junit4:junit4]   2> 4474 T180 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 4474 T180 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 4475 T180 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 4495 T180 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 4567 T224 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f5a6472 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 4572 T180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 4993 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4994 T202 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:44503_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:44503/solr"}
[junit4:junit4]   2> 4994 T202 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 5022 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5022 T219 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5076 T180 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 5076 T180 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:44503/solr shardId:shard1
[junit4:junit4]   2> 5078 T180 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 5106 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0003 type:delete cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 5108 T180 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 5108 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0003 type:create cxid:0x44 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5119 T180 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 5120 T180 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 5120 T180 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44503/solr/collection1/
[junit4:junit4]   2> 5120 T180 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 5121 T180 oasc.SyncStrategy.syncToMe http://127.0.0.1:44503/solr/collection1/ has no replicas
[junit4:junit4]   2> 5121 T180 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44503/solr/collection1/
[junit4:junit4]   2> 5121 T180 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 5134 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0003 type:create cxid:0x4e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5532 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5556 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5556 T219 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5603 T180 oasc.ZkController.register We are http://127.0.0.1:44503/solr/collection1/ and leader is http://127.0.0.1:44503/solr/collection1/
[junit4:junit4]   2> 5603 T180 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44503/solr
[junit4:junit4]   2> 5604 T180 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 5604 T180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 5607 T180 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5609 T180 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 5609 T180 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 5610 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 5802 T180 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 5805 T180 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:48463
[junit4:junit4]   2> 5806 T180 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 5807 T180 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222
[junit4:junit4]   2> 5807 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222/'
[junit4:junit4]   2> 5856 T180 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 5856 T180 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 5857 T180 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222
[junit4:junit4]   2> 5857 T180 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222/solr.xml
[junit4:junit4]   2> 5858 T180 oasc.CoreContainer.<init> New CoreContainer 1730644545
[junit4:junit4]   2> 5859 T180 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222/'
[junit4:junit4]   2> 5859 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222/'
[junit4:junit4]   2> 5895 T180 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 5912 T180 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:20217/solr
[junit4:junit4]   2> 5913 T180 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:20217/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@a5c6345
[junit4:junit4]   2> 5914 T234 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:20217
[junit4:junit4]   2> 5914 T180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 5916 T234 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:20217, initiating session
[junit4:junit4]   2> 5916 T182 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:51729
[junit4:junit4]   2> 5916 T182 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:51729
[junit4:junit4]   2> 5918 T184 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1113a0c0004 with negotiated timeout 5000 for client /140.211.11.196:51729
[junit4:junit4]   2> 5918 T234 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:20217, sessionid = 0x13ac1113a0c0004, negotiated timeout = 5000
[junit4:junit4]   2> 5919 T235 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a5c6345 name:ZooKeeperConnection Watcher:127.0.0.1:20217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 5919 T180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 5920 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5922 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5924 T180 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48463_solr
[junit4:junit4]   2> 5924 T185 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1113a0c0004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:48463_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:48463_solr
[junit4:junit4]   2> 5931 T180 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48463_solr
[junit4:junit4]   2> 5935 T219 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 5935 T201 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 5936 T180 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 5936 T180 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 5937 T180 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 5937 T180 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 5938 T180 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 5938 T180 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 5939 T180 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 5939 T180 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 5939 T180 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 5940 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 5963 T180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5970 T180 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222/collection1
[junit4:junit4]   2> 5970 T180 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5971 T180 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5972 T180 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5973 T180 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222/collection1/'
[junit4:junit4]   2> 5975 T180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty2-1351858868222/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5975 T180 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty2-1351858868222/collection1/lib/README' to classloader
[junit4:junit4]   2> 6031 T180 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6069 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6070 T202 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:44503_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:44503/solr"}
[junit4:junit4]   2> 6094 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6094 T219 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6094 T235 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6106 T180 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 6107 T180 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6118 T180 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6808 T180 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6821 T180 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6826 T180 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6851 T180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6858 T180 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6865 T180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6874 T180 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6876 T180 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty2-1351858868222/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty2/
[junit4:junit4]   2> 6877 T180 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13d380dc
[junit4:junit4]   2> 6879 T180 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty2/index/
[junit4:junit4]   2> 6880 T180 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6881 T180 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty2/index forceNew:false
[junit4:junit4]   2> 7005 T180 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty2/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7006 T180 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7006 T180 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7010 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7011 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7011 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7012 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7013 T180 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7013 T180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7014 T180 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7015 T180 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7016 T180 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 7016 T180 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 7017 T180 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 7018 T180 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 7019 T180 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 7019 T180 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 7020 T180 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 7021 T180 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7022 T180 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7022 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7023 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7024 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7025 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7026 T180 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7026 T180 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7027 T180 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 7028 T180 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 7041 T180 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7046 T180 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7048 T180 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 7052 T180 oass.SolrIndexSearcher.<init> Opening Searcher@2eba74df main
[junit4:junit4]   2> 7053 T180 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7054 T180 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7055 T180 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 7163 T180 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7343 T240 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2eba74df main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7351 T180 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7608 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7609 T202 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:48463_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:48463/solr"}
[junit4:junit4]   2> 7622 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7622 T235 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7622 T219 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7860 T180 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7860 T180 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:48463/solr shardId:shard1
[junit4:junit4]   2> 7865 T180 oasc.ZkController.register We are http://127.0.0.1:48463/solr/collection1/ and leader is http://127.0.0.1:44503/solr/collection1/
[junit4:junit4]   2> 7865 T180 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48463/solr
[junit4:junit4]   2> 7866 T180 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 7866 T180 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C20 name=collection1 org.apache.solr.core.SolrCore@28740b0f url=http://127.0.0.1:48463/solr/collection1 node=127.0.0.1:48463_solr
[junit4:junit4]   2> 7874 T241 C20 P48463 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 7874 T180 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7875 T241 C20 P48463 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 7875 T241 C20 P48463 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7880 T180 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 7881 T180 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7881 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7889 T241 C20 P48463 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:44503/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 7900 T241 C20 P48463 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 7903 T180 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:20217/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1a301110
[junit4:junit4]   2> 7904 T242 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:20217
[junit4:junit4]   2> 7904 T180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7905 T242 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:20217, initiating session
[junit4:junit4]   2> 7906 T182 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:64040
[junit4:junit4]   2> 7906 T182 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:64040
[junit4:junit4]   2> 7910 T184 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1113a0c0005 with negotiated timeout 10000 for client /140.211.11.196:64040
[junit4:junit4]   2> 7910 T242 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:20217, sessionid = 0x13ac1113a0c0005, negotiated timeout = 10000
[junit4:junit4]   2> 7910 T243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a301110 name:ZooKeeperConnection Watcher:127.0.0.1:20217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7911 T180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7912 T180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7917 T180 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 7918 T180 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 7922 T241 C20 P48463 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48463/solr START replicas=[http://127.0.0.1:44503/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 7931 T180 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7942 T241 C20 P48463 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 7944 T241 C20 P48463 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 7944 T241 C20 P48463 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 7944 T241 C20 P48463 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8128 T202 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8129 T202 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:48463_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:48463/solr"}
[junit4:junit4]   2> 8140 T243 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8140 T201 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8140 T235 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8140 T219 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> ASYNC  NEW_CORE C21 name=collection1 org.apache.solr.core.SolrCore@1adc811d url=http://127.0.0.1:44503/solr/collection1 node=127.0.0.1:44503_solr
[junit4:junit4]   2> 9224 T220 C21 P44503 REQ /get {distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=22 
[junit4:junit4]   2> ASYNC  NEW_CORE C22 name=collection1 org.apache.solr.core.SolrCore@286ee8d5 url=http://127.0.0.1:19836/solr/collection1 node=127.0.0.1:19836_solr
[junit4:junit4]   2> 9288 T207 C22 P19836 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16
[junit4:junit4]   2> 9335 T205 C22 P19836 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351858862598/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9336 T205 C22 P19836 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9344 T205 C22 P19836 /update {wt=javabin&version=2} {add=[1 (1417526768483434496)]} 0 74
[junit4:junit4]   2> 9344 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10001 (1417526768483434497)]} 0 74
[junit4:junit4]   2> 9395 T245 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:20217/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@58f19e9
[junit4:junit4]   2> 9396 T247 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:20217
[junit4:junit4]   2> 9396 T245 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 9397 T247 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:20217, initiating session
[junit4:junit4]   2> 9397 T182 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:21472
[junit4:junit4]   2> 9398 T182 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:21472
[junit4:junit4]   2> 9400 T184 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1113a0c0006 with negotiated timeout 10000 for client /140.211.11.196:21472
[junit4:junit4]   2> 9400 T247 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:20217, sessionid = 0x13ac1113a0c0006, negotiated timeout = 10000
[junit4:junit4]   2> 9400 T248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58f19e9 name:ZooKeeperConnection Watcher:127.0.0.1:20217/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 9401 T245 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 9401 T245 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9432 T220 C21 P44503 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty1/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9432 T220 C21 P44503 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2>  C20_STATE=coll:collection1 core:collection1 props:{shard=shard1, roles=null, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:48463_solr, base_url=http://127.0.0.1:48463/solr}
[junit4:junit4]   2> 9498 T237 C20 P48463 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351858862598/jetty2/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9499 T237 C20 P48463 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9507 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=14 
[junit4:junit4]   2> 9507 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=14 
[junit4:junit4]   2> 9508 T220 C21 P44503 /update {wt=javabin&version=2} {add=[1 (1417526768589340672)]} 0 89
[junit4:junit4]   2> 9508 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10001 (1417526768589340673)]} 0 89
[junit4:junit4]   2> 9516 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10002 (1417526768678469633)]} 0 3
[junit4:junit4]   2> 9516 T206 C22 P19836 /update {wt=javabin&version=2} {add=[2 (1417526768678469632)]} 0 4
[junit4:junit4]   2> 9530 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9531 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9531 T222 C21 P44503 /update {wt=javabin&version=2} {add=[2 (1417526768685809664)]} 0 11
[junit4:junit4]   2> 9535 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10002 (1417526768685809665)]} 0 15
[junit4:junit4]   2> 9539 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10001 (-1417526768705732608)]} 0 0
[junit4:junit4]   2> 9539 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[1 (-1417526768705732609)]} 0 0
[junit4:junit4]   2> 9552 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9553 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 9554 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[10001 (-1417526768710975488)]} 0 10
[junit4:junit4]   2> 9554 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[1 (-1417526768709926912)]} 0 11
[junit4:junit4]   2> 9561 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10003 (1417526768725655552)]} 0 3
[junit4:junit4]   2> 9561 T206 C22 P19836 /update {wt=javabin&version=2} {add=[3 (1417526768725655553)]} 0 3
[junit4:junit4]   2> 9573 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9574 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9575 T220 C21 P44503 /update {wt=javabin&version=2} {add=[3 (1417526768732995584)]} 0 10
[junit4:junit4]   2> 9575 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10003 (1417526768732995585)]} 0 10
[junit4:junit4]   2> 9579 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[2 (-1417526768747675648)]} 0 0
[junit4:junit4]   2> 9579 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10002 (-1417526768747675649)]} 0 0
[junit4:junit4]   2> 9587 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 9587 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9588 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[2 (-1417526768751869952)]} 0 5
[junit4:junit4]   2> 9588 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[10002 (-1417526768752918528)]} 0 5
[junit4:junit4]   2> 9594 T205 C22 P19836 /update {wt=javabin&version=2} {add=[4 (1417526768761307137)]} 0 2
[junit4:junit4]   2> 9596 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10004 (1417526768761307136)]} 0 4
[junit4:junit4]   2> 9605 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9606 T222 C21 P44503 /update {wt=javabin&version=2} {add=[4 (1417526768767598592)]} 0 8
[junit4:junit4]   2> 9610 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 9610 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[3 (-1417526768780181504)]} 0 0
[junit4:junit4]   2> 9611 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10004 (1417526768769695744)]} 0 11
[junit4:junit4]   2> 9615 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[10003 (-1417526768785424384)]} 0 0
[junit4:junit4]   2> 9617 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9619 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[3 (-1417526768784375808)]} 0 5
[junit4:junit4]   2> 9622 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9623 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[10003 (-1417526768789618688)]} 0 4
[junit4:junit4]   2> 9626 T206 C22 P19836 /update {wt=javabin&version=2} {add=[5 (1417526768793812992)]} 0 3
[junit4:junit4]   2> 9629 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10005 (1417526768798007296)]} 0 2
[junit4:junit4]   2> 9638 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9638 T223 C21 P44503 /update {wt=javabin&version=2} {add=[5 (1417526768801153024)]} 0 8
[junit4:junit4]   2> 9642 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[4 (-1417526768813735936)]} 0 0
[junit4:junit4]   2> 9644 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 9645 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10005 (1417526768804298752)]} 0 12
[junit4:junit4]   2> 9649 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9649 T206 C22 P19836 /update {wt=javabin&version=2} {delete=[10004 (-1417526768821075968)]} 0 0
[junit4:junit4]   2> 9650 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[4 (-1417526768817930240)]} 0 5
[junit4:junit4]   2> 9656 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9656 T207 C22 P19836 /update {wt=javabin&version=2} {add=[6 (1417526768826318848)]} 0 2
[junit4:junit4]   2> 9657 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[10004 (-1417526768825270272)]} 0 5
[junit4:junit4]   2> 9663 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10006 (1417526768833658880)]} 0 3
[junit4:junit4]   2> 9670 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9672 T220 C21 P44503 /update {wt=javabin&version=2} {add=[6 (1417526768832610304)]} 0 11
[junit4:junit4]   2> 9677 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9677 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10006 (1417526768840998912)]} 0 9
[junit4:junit4]   2> 9678 T206 C22 P19836 /update {wt=javabin&version=2} {add=[7 (1417526768849387520)]} 0 2
[junit4:junit4]   2> 9685 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10007 (1417526768855678976)]} 0 3
[junit4:junit4]   2> 9692 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9693 T223 C21 P44503 /update {wt=javabin&version=2} {add=[7 (1417526768856727552)]} 0 10
[junit4:junit4]   2> 9699 T205 C22 P19836 /update {wt=javabin&version=2} {add=[8 (1417526768871407616)]} 0 2
[junit4:junit4]   2> 9700 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 9701 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10007 (1417526768864067584)]} 0 12
[junit4:junit4]   2> 9711 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9712 T223 C21 P44503 /update {wt=javabin&version=2} {add=[8 (1417526768877699072)]} 0 9
[junit4:junit4]   2> 9716 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[6 (-1417526768891330560)]} 0 0
[junit4:junit4]   2> 9724 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9725 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[6 (-1417526768895524864)]} 0 5
[junit4:junit4]   2> 9734 T205 C22 P19836 /update {wt=javabin&version=2} {add=[9 (1417526768904962048)]} 0 5
[junit4:junit4]   2> 9779 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10008 (1417526768880844800)]} 0 73
[junit4:junit4]   2> 9791 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9792 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10008 (1417526768961585152)]} 0 9
[junit4:junit4]   2> 9796 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10006 (-1417526768975216640)]} 0 1
[junit4:junit4]   2> 9804 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 9805 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[10006 (-1417526768979410944)]} 0 5
[junit4:junit4]   2> 9811 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10009 (1417526768988848128)]} 0 2
[junit4:junit4]   2> 9825 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9826 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10009 (1417526768995139584)]} 0 11
[junit4:junit4]   2> 9834 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=58 
[junit4:junit4]   2> 9834 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10010 (1417526769011916800)]} 0 3
[junit4:junit4]   2> 9835 T222 C21 P44503 /update {wt=javabin&version=2} {add=[9 (1417526768915447808)]} 0 96
[junit4:junit4]   2> 9841 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10 (1417526769020305408)]} 0 2
[junit4:junit4]   2> 9847 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9848 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10010 (1417526769020305408)]} 0 9
[junit4:junit4]   2> 9854 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10011 (1417526769033936896)]} 0 2
[junit4:junit4]   2> 9856 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=6 
[junit4:junit4]   2> 9858 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10 (1417526769026596864)]} 0 13
[junit4:junit4]   2> 9864 T206 C22 P19836 /update {wt=javabin&version=2} {add=[11 (1417526769044422656)]} 0 2
[junit4:junit4]   2> 9876 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9877 T220 C21 P44503 /update {wt=javabin&version=2} {add=[11 (1417526769050714112)]} 0 9
[junit4:junit4]   2> 9881 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[7 (-1417526769064345600)]} 0 0
[junit4:junit4]   2> 9888 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9890 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[7 (-1417526769068539904)]} 0 5
[junit4:junit4]   2> 9897 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=32 
[junit4:junit4]   2> 9897 T205 C22 P19836 /update {wt=javabin&version=2} {add=[12 (1417526769077977088)]} 0 3
[junit4:junit4]   2> 9898 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10011 (1417526769041276928)]} 0 39
[junit4:junit4]   2> 9903 T206 C22 P19836 /update {wt=javabin&version=2} {delete=[10007 (-1417526769087414272)]} 0 0
[junit4:junit4]   2> 9911 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9912 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[10007 (-1417526769092657152)]} 0 4
[junit4:junit4]   2> 9918 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10012 (1417526769101045760)]} 0 2
[junit4:junit4]   2> 9933 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9934 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10012 (1417526769108385792)]} 0 11
[junit4:junit4]   2> 9937 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 9938 T220 C21 P44503 /update {wt=javabin&version=2} {add=[12 (1417526769086365696)]} 0 36
[junit4:junit4]   2> 9940 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10013 (1417526769124114432)]} 0 3
[junit4:junit4]   2> 9944 T205 C22 P19836 /update {wt=javabin&version=2} {add=[13 (1417526769128308736)]} 0 2
[junit4:junit4]   2> 9951 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9952 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10013 (1417526769130405888)]} 0 9
[junit4:junit4]   2> 9955 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9956 T222 C21 P44503 /update {wt=javabin&version=2} {add=[13 (1417526769134600192)]} 0 9
[junit4:junit4]   2> 9958 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10014 (1417526769142988800)]} 0 3
[junit4:junit4]   2> 9962 T207 C22 P19836 /update {wt=javabin&version=2} {add=[14 (1417526769147183104)]} 0 2
[junit4:junit4]   2> 9973 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9974 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10014 (1417526769149280256)]} 0 13
[junit4:junit4]   2> 9975 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9976 T223 C21 P44503 /update {wt=javabin&version=2} {add=[14 (1417526769154523136)]} 0 9
[junit4:junit4]   2> 9986 T205 C22 P19836 /update {wt=javabin&version=2} {add=[15 (1417526769169203200)]} 0 6
[junit4:junit4]   2> 9998 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9999 T222 C21 P44503 /update {wt=javabin&version=2} {add=[15 (1417526769178640384)]} 0 9
[junit4:junit4]   2> 10004 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10 (-1417526769193320448)]} 0 0
[junit4:junit4]   2> 10011 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10015 (1417526769167106048)]} 0 32
[junit4:junit4]   2> 10011 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10012 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[10 (-1417526769197514752)]} 0 4
[junit4:junit4]   2> 10019 T205 C22 P19836 /update {wt=javabin&version=2} {add=[16 (1417526769205903360)]} 0 3
[junit4:junit4]   2> 10023 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10023 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10015 (1417526769204854784)]} 0 9
[junit4:junit4]   2> 10027 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10010 (-1417526769217437696)]} 0 0
[junit4:junit4]   2> 10035 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10036 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[10010 (-1417526769221632000)]} 0 5
[junit4:junit4]   2> 10044 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10016 (1417526769232117760)]} 0 3
[junit4:junit4]   2> 10055 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10056 T222 C21 P44503 /update {wt=javabin&version=2} {add=[16 (1417526769213243392)]} 0 33
[junit4:junit4]   2> 10059 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10059 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10016 (1417526769239457792)]} 0 11
[junit4:junit4]   2> 10061 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[11 (-1417526769253089280)]} 0 0
[junit4:junit4]   2> 10063 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10011 (-1417526769255186432)]} 0 1
[junit4:junit4]   2> 10069 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10070 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10070 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[11 (-1417526769257283584)]} 0 5
[junit4:junit4]   2> 10071 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[10011 (-1417526769259380736)]} 0 4
[junit4:junit4]   2> 10076 T206 C22 P19836 /update {wt=javabin&version=2} {add=[17 (1417526769266720768)]} 0 3
[junit4:junit4]   2> 10078 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10017 (1417526769267769344)]} 0 3
[junit4:junit4]   2> 10091 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10092 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10017 (1417526769275109376)]} 0 10
[junit4:junit4]   2> 10096 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10015 (-1417526769289789440)]} 0 0
[junit4:junit4]   2> 10103 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10104 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[10015 (-1417526769293983744)]} 0 5
[junit4:junit4]   2> 10110 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10018 (1417526769302372352)]} 0 2
[junit4:junit4]   2> 10113 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=28 
[junit4:junit4]   2> 10114 T223 C21 P44503 /update {wt=javabin&version=2} {add=[17 (1417526769273012224)]} 0 35
[junit4:junit4]   2> 10118 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[15 (-1417526769312858112)]} 0 1
[junit4:junit4]   2> 10121 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10122 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10018 (1417526769308663808)]} 0 9
[junit4:junit4]   2> 10125 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10126 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[15 (-1417526769317052416)]} 0 5
[junit4:junit4]   2> 10132 T206 C22 P19836 /update {wt=javabin&version=2} {add=[18 (1417526769325441024)]} 0 2
[junit4:junit4]   2> 10146 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10147 T220 C21 P44503 /update {wt=javabin&version=2} {add=[18 (1417526769332781056)]} 0 11
[junit4:junit4]   2> 10153 T205 C22 P19836 /update {wt=javabin&version=2} {add=[19 (1417526769347461120)]} 0 2
[junit4:junit4]   2> 10164 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10165 T222 C21 P44503 /update {wt=javabin&version=2} {add=[19 (1417526769353752576)]} 0 9
[junit4:junit4]   2> 10166 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10019 (1417526769321246720)]} 0 40
[junit4:junit4]   2> 10170 T206 C22 P19836 /update {wt=javabin&version=2} {delete=[16 (-1417526769367384064)]} 0 0
[junit4:junit4]   2> 10178 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10179 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10179 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[16 (-1417526769372626944)]} 0 4
[junit4:junit4]   2> 10180 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10019 (1417526769368432640)]} 0 10
[junit4:junit4]   2> 10184 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[10016 (-1417526769382064128)]} 0 0
[junit4:junit4]   2> 10190 T207 C22 P19836 /update {wt=javabin&version=2} {add=[20 (1417526769382064129)]} 0 6
[junit4:junit4]   2> 10191 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10192 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[10016 (-1417526769386258432)]} 0 4
[junit4:junit4]   2> 10198 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10020 (1417526769394647040)]} 0 3
[junit4:junit4]   2> 10201 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10203 T223 C21 P44503 /update {wt=javabin&version=2} {add=[20 (1417526769392549888)]} 0 9
[junit4:junit4]   2> 10210 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10211 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10020 (1417526769400938496)]} 0 9
[junit4:junit4]   2> 10217 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10021 (1417526769414569984)]} 0 3
[junit4:junit4]   2> 10229 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10230 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10021 (1417526769421910016)]} 0 9
[junit4:junit4]   2> 10234 T205 C22 P19836 /update {wt=javabin&version=2} {add=[21 (1417526769406181376)]} 0 28
[junit4:junit4]   2> 10234 T206 C22 P19836 /update {wt=javabin&version=2} {delete=[10021 (-1417526769434492928)]} 0 1
[junit4:junit4]   2> 10241 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10242 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[10021 (-1417526769438687232)]} 0 5
[junit4:junit4]   2> 10245 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10246 T223 C21 P44503 /update {wt=javabin&version=2} {add=[21 (1417526769438687233)]} 0 9
[junit4:junit4]   2> 10248 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10022 (1417526769447075840)]} 0 3
[junit4:junit4]   2> 10250 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[21 (-1417526769451270144)]} 0 1
[junit4:junit4]   2> 10257 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10258 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[21 (-1417526769455464448)]} 0 5
[junit4:junit4]   2> 10259 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10259 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10022 (1417526769453367296)]} 0 8
[junit4:junit4]   2> 10263 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10022 (-1417526769464901632)]} 0 1
[junit4:junit4]   2> 10265 T206 C22 P19836 /update {wt=javabin&version=2} {add=[22 (1417526769463853056)]} 0 4
[junit4:junit4]   2> 10270 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10271 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[10022 (-1417526769469095936)]} 0 5
[junit4:junit4]   2> 10276 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10023 (1417526769476435968)]} 0 3
[junit4:junit4]   2> 10276 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10277 T222 C21 P44503 /update {wt=javabin&version=2} {add=[22 (1417526769471193088)]} 0 9
[junit4:junit4]   2> 10281 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[22 (-1417526769482727424)]} 0 1
[junit4:junit4]   2> 10287 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10288 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10023 (1417526769482727424)]} 0 9
[junit4:junit4]   2> 10288 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10289 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[22 (-1417526769487970304)]} 0 5
[junit4:junit4]   2> 10292 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10024 (1417526769494261760)]} 0 2
[junit4:junit4]   2> 10294 T205 C22 P19836 /update {wt=javabin&version=2} {add=[23 (1417526769496358912)]} 0 2
[junit4:junit4]   2> 10304 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10304 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10024 (1417526769499504640)]} 0 8
[junit4:junit4]   2> 10309 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10023 (-1417526769513136128)]} 0 0
[junit4:junit4]   2> 10316 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10317 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[10023 (-1417526769517330432)]} 0 5
[junit4:junit4]   2> 10323 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10025 (1417526769525719040)]} 0 3
[junit4:junit4]   2> 10328 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=26 
[junit4:junit4]   2> 10329 T220 C21 P44503 /update {wt=javabin&version=2} {add=[23 (1417526769501601792)]} 0 32
[junit4:junit4]   2> 10335 T207 C22 P19836 /update {wt=javabin&version=2} {add=[24 (1417526769538301952)]} 0 3
[junit4:junit4]   2> 10348 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10349 T223 C21 P44503 /update {wt=javabin&version=2} {add=[24 (1417526769544593408)]} 0 11
[junit4:junit4]   2> 10353 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[23 (-1417526769559273472)]} 0 0
[junit4:junit4]   2> 10360 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10361 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[23 (-1417526769563467776)]} 0 4
[junit4:junit4]   2> 10364 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10365 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10025 (1417526769532010496)]} 0 39
[junit4:junit4]   2> 10367 T206 C22 P19836 /update {wt=javabin&version=2} {add=[25 (1417526769571856384)]} 0 3
[junit4:junit4]   2> 10369 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[10024 (-1417526769576050688)]} 0 1
[junit4:junit4]   2> 10376 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10377 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[10024 (-1417526769580244992)]} 0 5
[junit4:junit4]   2> 10377 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10378 T223 C21 P44503 /update {wt=javabin&version=2} {add=[25 (1417526769577099264)]} 0 9
[junit4:junit4]   2> 10382 T205 C22 P19836 /update {wt=javabin&version=2} {delete=[24 (-1417526769589682176)]} 0 1
[junit4:junit4]   2> 10383 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10026 (1417526769588633600)]} 0 2
[junit4:junit4]   2> 10389 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10390 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[24 (-1417526769593876480)]} 0 4
[junit4:junit4]   2> 10393 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10394 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10026 (1417526769593876481)]} 0 8
[junit4:junit4]   2> 10395 T207 C22 P19836 /update {wt=javabin&version=2} {add=[26 (1417526769602265088)]} 0 2
[junit4:junit4]   2> 10399 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10027 (1417526769606459392)]} 0 2
[junit4:junit4]   2> 10410 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10410 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10027 (1417526769611702272)]} 0 7
[junit4:junit4]   2> 10417 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10028 (1417526769624285184)]} 0 3
[junit4:junit4]   2> 10427 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10428 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10028 (1417526769630576640)]} 0 8
[junit4:junit4]   2> 10434 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=30 
[junit4:junit4]   2> 10434 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10029 (1417526769642110976)]} 0 3
[junit4:junit4]   2> 10435 T223 C21 P44503 /update {wt=javabin&version=2} {add=[26 (1417526769608556544)]} 0 35
[junit4:junit4]   2> 10470 T205 C22 P19836 /update {wt=javabin&version=2} {add=[27 (1417526769648402432)]} 0 33
[junit4:junit4]   2> 10477 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10478 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10029 (1417526769648402432)]} 0 41
[junit4:junit4]   2> 10482 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10483 T222 C21 P44503 /update {wt=javabin&version=2} {add=[27 (1417526769687199744)]} 0 8
[junit4:junit4]   2> 10485 T206 C22 P19836 /update {wt=javabin&version=2} {add=[10030 (1417526769694539776)]} 0 4
[junit4:junit4]   2> 10490 T207 C22 P19836 /update {wt=javabin&version=2} {add=[28 (1417526769699782656)]} 0 4
[junit4:junit4]   2> 10498 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10499 T223 C21 P44503 /update {wt=javabin&version=2} {add=[10030 (1417526769701879808)]} 0 11
[junit4:junit4]   2> 10503 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10503 T220 C21 P44503 /update {wt=javabin&version=2} {add=[28 (1417526769707122688)]} 0 10
[junit4:junit4]   2> 10505 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10031 (1417526769716559872)]} 0 3
[junit4:junit4]   2> 10508 T206 C22 P19836 /update {wt=javabin&version=2} {add=[29 (1417526769720754176)]} 0 2
[junit4:junit4]   2> 10517 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10518 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10031 (1417526769722851328)]} 0 9
[junit4:junit4]   2> 10519 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10519 T220 C21 P44503 /update {wt=javabin&version=2} {add=[29 (1417526769725997056)]} 0 8
[junit4:junit4]   2> 10524 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10032 (1417526769736482816)]} 0 3
[junit4:junit4]   2> 10524 T205 C22 P19836 /update {wt=javabin&version=2} {add=[30 (1417526769737531392)]} 0 2
[junit4:junit4]   2> 10535 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10536 T220 C21 P44503 /update {wt=javabin&version=2} {add=[30 (1417526769742774273)]} 0 9
[junit4:junit4]   2> 10542 T206 C22 P19836 /update {wt=javabin&version=2} {add=[31 (1417526769755357184)]} 0 3
[junit4:junit4]   2> 10561 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=28 
[junit4:junit4]   2> 10563 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10032 (1417526769742774272)]} 0 36
[junit4:junit4]   2> 10569 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10033 (1417526769783668736)]} 0 2
[junit4:junit4]   2> 10577 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10578 T223 C21 P44503 /update {wt=javabin&version=2} {add=[31 (1417526769761648640)]} 0 33
[junit4:junit4]   2> 10582 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10583 T220 C21 P44503 /update {wt=javabin&version=2} {add=[10033 (1417526769789960192)]} 0 11
[junit4:junit4]   2> 10584 T205 C22 P19836 /update {wt=javabin&version=2} {add=[32 (1417526769799397376)]} 0 3
[junit4:junit4]   2> 10587 T206 C22 P19836 /update {wt=javabin&version=2} {delete=[10028 (-1417526769804640256)]} 0 1
[junit4:junit4]   2> 10594 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10595 T223 C21 P44503 /update {wt=javabin&version=2} {delete=[10028 (-1417526769808834560)]} 0 5
[junit4:junit4]   2> 10597 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10598 T222 C21 P44503 /update {wt=javabin&version=2} {add=[32 (1417526769805688832)]} 0 11
[junit4:junit4]   2> 10604 T205 C22 P19836 /update {wt=javabin&version=2} {add=[33 (1417526769820368896)]} 0 3
[junit4:junit4]   2> 10617 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10618 T223 C21 P44503 /update {wt=javabin&version=2} {add=[33 (1417526769827708928)]} 0 9
[junit4:junit4]   2> 10622 T206 C22 P19836 /update {wt=javabin&version=2} {delete=[28 (-1417526769841340416)]} 0 0
[junit4:junit4]   2> 10628 T207 C22 P19836 /update {wt=javabin&version=2} {add=[10034 (1417526769817223168)]} 0 29
[junit4:junit4]   2> 10629 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10630 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[28 (-1417526769845534720)]} 0 5
[junit4:junit4]   2> 10636 T205 C22 P19836 /update {wt=javabin&version=2} {add=[34 (1417526769853923328)]} 0 3
[junit4:junit4]   2> 10639 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10640 T222 C21 P44503 /update {wt=javabin&version=2} {add=[10034 (1417526769851826176)]} 0 9
[junit4:junit4]   2> 10644 T206 C22 P19836 /update {wt=javabin&version=2} {delete=[10029 (-1417526769864409088)]} 0 1
[junit4:junit4]   2> 10647 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10648 T223 C21 P44503 /update {wt=javabin&version=2} {add=[34 (1417526769860214784)]} 0 8
[junit4:junit4]   2> 10652 T207 C22 P19836 /update {wt=javabin&version=2} {delete=[29 (-1417526769872797696)]} 0 1
[junit4:junit4]   2> 10654 T237 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10655 T220 C21 P44503 /update {wt=javabin&version=2} {delete=[10029 (-1417526769870700544)]} 0 5
[junit4:junit4]   2> 10659 T239 C20 P48463 REQ /update {distrib.from=http://127.0.0.1:44503/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10660 T222 C21 P44503 /update {wt=javabin&version=2} {delete=[29 (-1417526769876992000)]} 0 5
[junit4:junit4]   2> 10661 T205 C22 P19836 /update {wt=javabin&version=2} {add=[10035 (1417526769880137728)]} 0 2
[junit4:junit4]   2> 10673 T238 C20 P48463 REQ /update {distrib.from=http://127.0.0

[...truncated too long message...]

ctChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    3) Thread[id=80, name=qtp1635915482-80 Acceptor0 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    4) Thread[id=81, name=qtp1635915482-81 Acceptor1 SelectChannelConnector@0.0.0.0:48884, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    5) Thread[id=82, name=qtp1635915482-82 Acceptor2 SelectChannelConnector@0.0.0.0:48884, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    6) Thread[id=99, name=qtp370838838-99 Acceptor1 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    7) Thread[id=98, name=qtp370838838-98 Acceptor0 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    8) Thread[id=31, name=qtp1013516039-31 Acceptor1 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >    9) Thread[id=30, name=qtp1013516039-30 Acceptor0 SelectChannelConnector@0.0.0.0:44571, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   10) Thread[id=32, name=qtp1013516039-32 Acceptor2 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   11) Thread[id=33, name=qtp1013516039-33 Acceptor3 SelectChannelConnector@0.0.0.0:44571, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   12) Thread[id=101, name=qtp370838838-101 Acceptor3 SelectChannelConnector@0.0.0.0:41359, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([97D4AB69C437B5E3]:0)
[junit4:junit4] Completed on J0 in 291.79s, 1 test, 2 errors <<< FAILURES!

[...truncated 180 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 1 error, 825 ignored

Total time: 21 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message