lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 833 - Still Failing
Date Sat, 03 Nov 2012 01:10:05 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/833/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:     1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest: 
   1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.CacheHeaderTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest:     1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.CacheHeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)




Build Log:
[...truncated 7635 lines...]
[junit4:junit4] Suite: org.apache.solr.servlet.CacheHeaderTest
[junit4:junit4]   2> 0 T13 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-CacheHeaderTest-1351904893286
[junit4:junit4]   2> 2 T13 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 532 T13 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 808 T13 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:30399
[junit4:junit4]   2> 862 T13 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 863 T13 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 863 T13 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: 'solr/'
[junit4:junit4]   2> 911 T13 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 911 T13 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 912 T13 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 912 T13 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr/solr.xml
[junit4:junit4]   2> 913 T13 oasc.CoreContainer.<init> New CoreContainer 1010923728
[junit4:junit4]   2> 913 T13 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 914 T13 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'solr/'
[junit4:junit4]   2> 914 T13 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/'
[junit4:junit4]   2> 954 T13 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1013 T13 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: solr/collection1
[junit4:junit4]   2> 1014 T13 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/collection1/'
[junit4:junit4]   2> 1139 T13 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1223 T13 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1227 T13 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1265 T13 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1821 T13 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1836 T13 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1841 T13 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1876 T13 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1886 T13 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1890 T13 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1895 T13 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1898 T13 oasc.SolrCore.<init> [collection1] Opening new SolrCore at solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/
[junit4:junit4]   2> 1899 T13 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79fcf790
[junit4:junit4]   2> 1900 T13 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/index/
[junit4:junit4]   2> 1901 T13 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1902 T13 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/index forceNew:false
[junit4:junit4]   2> 1904 T13 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1904 T13 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1905 T13 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 1907 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1908 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 1909 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1909 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 1910 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1910 T13 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1911 T13 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1912 T13 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1913 T13 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1914 T13 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 1914 T13 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 1915 T13 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1916 T13 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 1916 T13 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 1917 T13 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 1918 T13 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1919 T13 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1919 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1920 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1921 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1922 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1922 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1924 T13 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1924 T13 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 1925 T13 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 1930 T13 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1936 T13 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1937 T13 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: []
[junit4:junit4]   2> 1942 T13 oass.SolrIndexSearcher.<init> Opening Searcher@1535d18b main
[junit4:junit4]   2> 1943 T13 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1944 T13 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1944 T13 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1946 T13 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 1972 T13 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2018 T13 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2018 T13 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2019 T13 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2019 T13 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2019 T13 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2020 T13 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2020 T13 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2021 T13 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2021 T13 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2022 T13 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2047 T28 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1535d18b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2053 T13 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2053 T13 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2054 T13 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2054 T13 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#30399
[junit4:junit4]   2> 2062 T13 oas.SolrTestCaseJ4.setUp ###Starting testLastModified
[junit4:junit4]   2> 2085 T13 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@66e90097
[junit4:junit4]   2> 3226 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=310 
[junit4:junit4]   2> 3466 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3477 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3481 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3487 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3494 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3494 T13 oas.SolrTestCaseJ4.tearDown ###Ending testLastModified
[junit4:junit4]   2> 3504 T13 oas.SolrTestCaseJ4.setUp ###Starting testCacheVetoHandler
[junit4:junit4]   2> 3568 T27 C0 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3569 T27 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3787 T27 C0 UPDATE [collection1] webapp=/solr path=/update params={stream.contentType=text/csv&stream.file=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/cachetest_csv1500715007tmp} {add=[100, 101, 102]} 0 271
[junit4:junit4]   2> 3789 T13 oas.SolrTestCaseJ4.tearDown ###Ending testCacheVetoHandler
[junit4:junit4]   2> 3799 T13 oas.SolrTestCaseJ4.setUp ###Starting testEtag
[junit4:junit4]   2> 3802 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3809 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=4 
[junit4:junit4]   2> 3820 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3823 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3827 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3830 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3846 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3849 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3850 T13 oas.SolrTestCaseJ4.tearDown ###Ending testEtag
[junit4:junit4]   2> 3858 T13 oas.SolrTestCaseJ4.setUp ###Starting testCacheControl
[junit4:junit4]   2> 3862 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3866 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3873 T27 C0 REQ [collection1] webapp=/solr path=/select params={q=solr&qt=standard} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3875 T13 oas.SolrTestCaseJ4.tearDown ###Ending testCacheControl
[junit4:junit4]   2> 3883 T13 oas.SolrTestCaseJ4.setUp ###Starting testCacheVetoException
[junit4:junit4]   2> 3888 T25 C0 oasc.SolrException.log Ignoring exception matching ignore_exception
[junit4:junit4]   2> 3888 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=xyz_ignore_exception:solr} status=400 QTime=2 
[junit4:junit4]   2> 3889 T13 oas.SolrTestCaseJ4.tearDown ###Ending testCacheVetoException
[junit4:junit4]   2> 3937 T13 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1010923728
[junit4:junit4]   2> 3938 T13 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@66e90097
[junit4:junit4]   2> 3959 T13 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=3,adds=3,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 3960 T13 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 3961 T13 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 3961 T13 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 4366 T13 C0 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _0_TestBloomFilteredLucene41Postings_0.tim, _0_nrm.cfs, _0_TestBloomFilteredLucene41Postings_0.tip, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0_nrm.cfe, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, segments_2, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 4367 T13 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4368 T13 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 4638 T13 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 4738 T13 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 4739 T13 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 4750 T12 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 24846 T12 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
[junit4:junit4]   2> 	   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 24847 T12 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 27892 T12 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), id=Pulsing41(freqCutoff=15 minBlockSize=49 maxBlockSize=138), range_facet_sl=Pulsing41(freqCutoff=15 minBlockSize=49 maxBlockSize=138), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), intDefault=PostingsFormat(name=Direct)}, sim=DefaultSimilarity, locale=sq, timezone=Atlantic/Jan_Mayen
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=477797528,total=492896256
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SearchHandlerTest, CacheHeaderTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CacheHeaderTest -Dtests.seed=2E9D6974F3019263 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sq -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR   0.00s J0 | CacheHeaderTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
[junit4:junit4]    >    1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)
[junit4:junit4] Completed on J0 in 28.13s, 5 tests, 2 errors <<< FAILURES!

[...truncated 520 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4:junit4]   2> 5 T65 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1351904922176
[junit4:junit4]   2> 7 T65 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 9 T66 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 10 T66 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 11 T66 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 108 T65 oasc.ZkTestServer.run start zk server on port:14721
[junit4:junit4]   2> 108 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@189ddde9
[junit4:junit4]   2> 109 T71 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 109 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 110 T71 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 111 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:63028
[junit4:junit4]   2> 112 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:63028
[junit4:junit4]   2> 112 T69 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 115 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460000 with negotiated timeout 10000 for client /140.211.11.196:63028
[junit4:junit4]   2> 116 T71 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460000, negotiated timeout = 10000
[junit4:junit4]   2> 116 T72 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@189ddde9 name:ZooKeeperConnection Watcher:127.0.0.1:14721 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 117 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 117 T65 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 122 T70 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac3d00a460000
[junit4:junit4]   2> 128 T72 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 129 T67 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:63028 which had sessionid 0x13ac3d00a460000
[junit4:junit4]   2> 128 T65 oaz.ZooKeeper.close Session: 0x13ac3d00a460000 closed
[junit4:junit4]   2> 130 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@27573872
[junit4:junit4]   2> 131 T73 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 131 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 132 T73 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 136 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:52262
[junit4:junit4]   2> 137 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:52262
[junit4:junit4]   2> 139 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460001 with negotiated timeout 10000 for client /140.211.11.196:52262
[junit4:junit4]   2> 139 T73 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460001, negotiated timeout = 10000
[junit4:junit4]   2> 140 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27573872 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 140 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 151 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 158 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 162 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 165 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 169 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 189 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 203 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 204 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 221 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 222 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 226 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 227 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 232 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 233 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 237 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 238 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 242 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 243 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 247 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 248 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 252 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 253 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 257 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 258 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 262 T70 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac3d00a460001
[junit4:junit4]   2> 263 T74 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 263 T65 oaz.ZooKeeper.close Session: 0x13ac3d00a460001 closed
[junit4:junit4]   2> 263 T67 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:52262 which had sessionid 0x13ac3d00a460001
[junit4:junit4]   2> 848 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1116 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:38930
[junit4:junit4]   2> 1157 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1158 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1158 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1191 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1192 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1192 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1192 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1193 T65 oasc.CoreContainer.<init> New CoreContainer 1814681656
[junit4:junit4]   2> 1194 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1194 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1229 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1243 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 1265 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@77a477b7
[junit4:junit4]   2> 1266 T85 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 1266 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1268 T85 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 1268 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:32429
[junit4:junit4]   2> 1269 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:32429
[junit4:junit4]   2> 1272 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460002 with negotiated timeout 5000 for client /140.211.11.196:32429
[junit4:junit4]   2> 1272 T85 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460002, negotiated timeout = 5000
[junit4:junit4]   2> 1273 T86 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77a477b7 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1273 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1286 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1295 T65 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1298 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38930_solr
[junit4:junit4]   2> 1302 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:38930_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:38930_solr
[junit4:junit4]   2> 1304 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38930_solr
[junit4:junit4]   2> 1308 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1308 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1309 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1309 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1309 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1310 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1310 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1311 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1311 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1312 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1324 T65 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1335 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1342 T65 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1346 T65 oasc.Overseer.start Overseer (id=88598440980119554-127.0.0.1:38930_solr-n_0000000000) starting
[junit4:junit4]   2> 1347 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1357 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1359 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1364 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1370 T88 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1370 T65 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1376 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1406 T87 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1446 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1447 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 1448 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1449 T65 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 1451 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1452 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1453 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1587 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1680 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1682 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1721 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2249 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2261 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2264 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2288 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2293 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2298 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2301 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2303 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/
[junit4:junit4]   2> 2307 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 2331 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index/
[junit4:junit4]   2> 2332 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2333 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index forceNew:false
[junit4:junit4]   2> 2342 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@762589c3; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2343 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2344 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2346 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2347 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2347 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2349 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2349 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2350 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2350 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2351 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2352 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2353 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2353 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2354 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2355 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2355 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2356 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2357 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2357 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2358 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2359 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2359 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2360 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2361 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2362 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2363 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2363 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2368 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2372 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2374 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 2378 T65 oass.SolrIndexSearcher.<init> Opening Searcher@201ba640 main
[junit4:junit4]   2> 2396 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2396 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2397 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2415 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2470 T93 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@201ba640 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2478 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2479 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x56 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2483 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2484 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:38930/solr shardId:control_shard
[junit4:junit4]   2> 2495 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 2509 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 2529 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:delete cxid:0x65 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2530 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2531 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x66 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2915 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2917 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:38930_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:38930/solr"}
[junit4:junit4]   2> 2926 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x6d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2954 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3035 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 3036 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 3037 T65 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38930/solr/collection1/
[junit4:junit4]   2> 3038 T65 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 3038 T65 oasc.SyncStrategy.syncToMe http://127.0.0.1:38930/solr/collection1/ has no replicas
[junit4:junit4]   2> 3039 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38930/solr/collection1/
[junit4:junit4]   2> 3039 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 3047 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x8b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3465 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3514 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3567 T65 oasc.ZkController.register We are http://127.0.0.1:38930/solr/collection1/ and leader is http://127.0.0.1:38930/solr/collection1/
[junit4:junit4]   2> 3571 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38930/solr
[junit4:junit4]   2> 3572 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3572 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3575 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3577 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3577 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3647 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3833 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3836 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:13518
[junit4:junit4]   2> 3837 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3838 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841
[junit4:junit4]   2> 3838 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/'
[junit4:junit4]   2> 3875 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3875 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3876 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841
[junit4:junit4]   2> 3877 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/solr.xml
[junit4:junit4]   2> 3877 T65 oasc.CoreContainer.<init> New CoreContainer 428806951
[junit4:junit4]   2> 3878 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/'
[junit4:junit4]   2> 3878 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/'
[junit4:junit4]   2> 3906 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3924 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 3925 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@6ffe8714
[junit4:junit4]   2> 3926 T103 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 3926 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3928 T103 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 3928 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:44255
[junit4:junit4]   2> 3929 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:44255
[junit4:junit4]   2> 4089 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460003 with negotiated timeout 5000 for client /140.211.11.196:44255
[junit4:junit4]   2> 4089 T103 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460003, negotiated timeout = 5000
[junit4:junit4]   2> 4090 T104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ffe8714 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4090 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4091 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4094 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4104 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4104 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13518_solr
[junit4:junit4]   2> 4105 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:38930_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:38930/solr"}
[junit4:junit4]   2> 4105 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13518_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13518_solr
[junit4:junit4]   2> 4107 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13518_solr
[junit4:junit4]   2> 4120 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4120 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4121 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4121 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4122 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4122 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4123 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4124 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4124 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4125 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4143 T86 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4144 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 4151 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 4159 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/collection1
[junit4:junit4]   2> 4160 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 4161 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 4162 T65 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 4164 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/collection1/'
[junit4:junit4]   2> 4166 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351904925841/collection1/lib/README' to classloader
[junit4:junit4]   2> 4167 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351904925841/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4234 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4313 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4315 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4345 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5006 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5017 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5020 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5044 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5049 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5053 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5055 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5058 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/
[junit4:junit4]   2> 5059 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 5060 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index/
[junit4:junit4]   2> 5060 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5061 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index forceNew:false
[junit4:junit4]   2> 5067 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5abe753a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5068 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5068 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 5071 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5071 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 5072 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5072 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 5074 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5074 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5075 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5075 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5076 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5077 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 5078 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 5078 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5079 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 5080 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 5080 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 5081 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5082 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5083 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5083 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5084 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5085 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5086 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5086 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5087 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 5088 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 5094 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5100 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5102 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 5106 T65 oass.SolrIndexSearcher.<init> Opening Searcher@22a79c31 main
[junit4:junit4]   2> 5107 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5107 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5108 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 5114 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5174 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22a79c31 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5185 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 5650 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5652 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13518_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13518/solr"}
[junit4:junit4]   2> 5652 T87 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 5743 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5743 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6189 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 6190 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:13518/solr shardId:shard1
[junit4:junit4]   2> 6191 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 6214 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 6215 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 6216 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6225 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 6226 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 6227 T65 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 6227 T65 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 6228 T65 oasc.SyncStrategy.syncToMe http://127.0.0.1:13518/solr/collection1/ has no replicas
[junit4:junit4]   2> 6228 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 6229 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 6235 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x4d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6249 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6267 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6267 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6295 T65 oasc.ZkController.register We are http://127.0.0.1:13518/solr/collection1/ and leader is http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 6295 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13518/solr
[junit4:junit4]   2> 6296 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 6296 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6299 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6300 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 6301 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6302 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 6472 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 6475 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:51722
[junit4:junit4]   2> 6477 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6477 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487
[junit4:junit4]   2> 6478 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/'
[junit4:junit4]   2> 6523 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 6523 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6524 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487
[junit4:junit4]   2> 6525 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/solr.xml
[junit4:junit4]   2> 6525 T65 oasc.CoreContainer.<init> New CoreContainer 845986605
[junit4:junit4]   2> 6526 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/'
[junit4:junit4]   2> 6527 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/'
[junit4:junit4]   2> 6553 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 6570 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 6571 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@690463c3
[junit4:junit4]   2> 6572 T119 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 6572 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6573 T119 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 6574 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:43782
[junit4:junit4]   2> 6574 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:43782
[junit4:junit4]   2> 6577 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460004 with negotiated timeout 5000 for client /140.211.11.196:43782
[junit4:junit4]   2> 6577 T119 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460004, negotiated timeout = 5000
[junit4:junit4]   2> 6578 T120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@690463c3 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6578 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6579 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6594 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6596 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51722_solr
[junit4:junit4]   2> 6597 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:51722_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:51722_solr
[junit4:junit4]   2> 6603 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51722_solr
[junit4:junit4]   2> 6607 T86 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6607 T104 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6608 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 6608 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 6609 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 6610 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 6610 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 6611 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 6611 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 6612 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 6612 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 6613 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 6632 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6640 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/collection1
[junit4:junit4]   2> 6641 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 6642 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 6642 T65 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 6644 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/collection1/'
[junit4:junit4]   2> 6646 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351904928487/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 6647 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351904928487/collection1/lib/README' to classloader
[junit4:junit4]   2> 6709 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6776 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6777 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13518_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13518/solr"}
[junit4:junit4]   2> 6789 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 6791 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6791 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6791 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6791 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6814 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 7498 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 7509 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7512 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7533 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7538 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7542 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7544 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7546 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/
[junit4:junit4]   2> 7547 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 7548 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index/
[junit4:junit4]   2> 7549 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7549 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index forceNew:false
[junit4:junit4]   2> 7559 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@71c0d0a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7559 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7560 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7562 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7563 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7564 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7564 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7565 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7565 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7566 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7566 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7567 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 7568 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 7569 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 7569 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 7570 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 7571 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 7571 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 7572 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7573 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7573 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7574 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7575 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7576 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7576 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7577 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7578 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 7579 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 7584 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7587 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7589 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 7592 T65 oass.SolrIndexSearcher.<init> Opening Searcher@67219b8c main
[junit4:junit4]   2> 7593 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7594 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7594 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 7600 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7650 T125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67219b8c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7658 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7801 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7802 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:51722_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:51722/solr"}
[junit4:junit4]   2> 7823 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7823 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7823 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8161 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 8161 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:51722/solr shardId:shard2
[junit4:junit4]   2> 8163 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 8179 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 8180 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 8181 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8191 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 8192 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 8193 T65 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51722/solr/collection1/
[junit4:junit4]   2> 8193 T65 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 8194 T65 oasc.SyncStrategy.syncToMe http://127.0.0.1:51722/solr/collection1/ has no replicas
[junit4:junit4]   2> 8194 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51722/solr/collection1/
[junit4:junit4]   2> 8195 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 8199 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x4c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8329 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8347 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8347 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8347 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8355 T65 oasc.ZkController.register We are http://127.0.0.1:51722/solr/collection1/ and leader is http://127.0.0.1:51722/solr/collection1/
[junit4:junit4]   2> 8355 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51722/solr
[junit4:junit4]   2> 8355 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8356 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8358 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8359 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8359 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8360 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8483 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 8487 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:50985
[junit4:junit4]   2> 8488 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8488 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540
[junit4:junit4]   2> 8489 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/'
[junit4:junit4]   2> 8525 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8526 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8526 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540
[junit4:junit4]   2> 8527 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/solr.xml
[junit4:junit4]   2> 8528 T65 oasc.CoreContainer.<init> New CoreContainer 1463607684
[junit4:junit4]   2> 8528 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/'
[junit4:junit4]   2> 8529 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/'
[junit4:junit4]   2> 8555 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8572 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 8573 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@3b39d369
[junit4:junit4]   2> 8574 T135 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 8574 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8575 T135 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 8576 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:27224
[junit4:junit4]   2> 8576 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:27224
[junit4:junit4]   2> 8579 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460005 with negotiated timeout 5000 for client /140.211.11.196:27224
[junit4:junit4]   2> 8579 T135 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460005, negotiated timeout = 5000
[junit4:junit4]   2> 8579 T136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b39d369 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8580 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8581 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460005 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8583 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460005 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8593 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50985_solr
[junit4:junit4]   2> 8594 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:50985_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:50985_solr
[junit4:junit4]   2> 8595 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50985_solr
[junit4:junit4]   2> 8599 T104 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8600 T86 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8600 T120 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8600 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 8601 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8601 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 8602 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8602 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8603 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8603 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8604 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8605 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8605 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 8625 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 8633 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/collection1
[junit4:junit4]   2> 8633 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 8634 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 8635 T65 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 8637 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/collection1/'
[junit4:junit4]   2> 8638 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty3-1351904930540/collection1/lib/README' to classloader
[junit4:junit4]   2> 8639 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty3-1351904930540/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 8700 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 8776 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 8778 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 8801 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 8862 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8864 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:51722_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:51722/solr"}
[junit4:junit4]   2> 8871 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8871 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8871 T136 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8871 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9461 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 9471 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 9474 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 9494 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9498 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9502 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9504 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9506 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/
[junit4:junit4]   2> 9507 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 9508 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index/
[junit4:junit4]   2> 9509 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 9510 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index forceNew:false
[junit4:junit4]   2> 9515 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c5e43ee; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9516 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9517 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 9520 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 9520 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 9521 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 9521 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 9522 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 9522 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 9523 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 9524 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 9524 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 9525 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 9526 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 9527 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 9527 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 9528 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 9528 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 9529 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 9530 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9530 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9531 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9532 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9532 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9533 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9534 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9534 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 9535 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 9540 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 9543 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 9545 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 9548 T65 oass.SolrIndexSearcher.<init> Opening Searcher@72cd429b main
[junit4:junit4]   2> 9549 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 9549 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 9550 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 9556 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 9610 T141 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72cd429b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 9617 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 9885 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9886 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50985_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50985/solr"}
[junit4:junit4]   2> 9900 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9900 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9900 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9900 T136 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10120 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 10120 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:50985/solr shardId:shard1
[junit4:junit4]   2> 10125 T65 oasc.ZkController.register We are http://127.0.0.1:50985/solr/collection1/ and leader is http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 10125 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50985/solr
[junit4:junit4]   2> 10126 T65 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 10127 T65 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@58a40787 url=http://127.0.0.1:50985/solr/collection1 node=127.0.0.1:50985_solr
[junit4:junit4]   2> 10134 T142 C2 P50985 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 10134 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10134 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 10135 T142 C2 P50985 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10136 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 10137 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 10137 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10137 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:13518/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 10145 T142 C2 P50985 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 10163 T142 C2 P50985 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50985/solr START replicas=[http://127.0.0.1:13518/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 10177 T142 C2 P50985 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 10178 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 10179 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 10179 T142 C2 P50985 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10337 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 10341 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:12275
[junit4:junit4]   2> 10342 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 10342 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326
[junit4:junit4]   2> 10343 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/'
[junit4:junit4]   2> 10406 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10407 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50985_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50985/solr"}
[junit4:junit4]   2> 10423 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10423 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10423 T136 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10423 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10517 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 10518 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 10519 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326
[junit4:junit4]   2> 10519 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/solr.xml
[junit4:junit4]   2> 10520 T65 oasc.CoreContainer.<init> New CoreContainer 1292438614
[junit4:junit4]   2> 10521 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/'
[junit4:junit4]   2> 10522 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/'
[junit4:junit4]   2> 10869 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 10984 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 10985 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d1cf4e5
[junit4:junit4]   2> 10986 T156 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 10986 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 10987 T156 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 10988 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:63117
[junit4:junit4]   2> 10988 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:63117
[junit4:junit4]   2> 10991 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460006 with negotiated timeout 5000 for client /140.211.11.196:63117
[junit4:junit4]   2> 10991 T156 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460006, negotiated timeout = 5000
[junit4:junit4]   2> 10991 T157 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d1cf4e5 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 10992 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 10993 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460006 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 10995 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460006 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11006 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:12275_solr
[junit4:junit4]   2> 11007 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460006 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:12275_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:12275_solr
[junit4:junit4]   2> 11008 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:12275_solr
[junit4:junit4]   2> 11012 T104 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11012 T120 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11012 T86 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11012 T136 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11013 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 11014 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 11014 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 11015 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 11016 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 11017 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 11017 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 11018 T65 oashc.HttpShardHandle

[...truncated too long message...]

dThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)
[junit4:junit4] Completed on J1 in 75.40s, 1 test, 2 errors <<< FAILURES!

[...truncated 170 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 861 ignored

Total time: 19 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message