lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-java7 - Build # 610 - Still Failing
Date Sat, 03 Nov 2012 16:20:52 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-java7/610/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=27, name=qtp806943770-27 Acceptor0 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=30, name=qtp806943770-30 Acceptor3 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=104, name=qtp1442633686-104 Acceptor1 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=105, name=qtp1442633686-105 Acceptor2 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=106, name=qtp1442633686-106 Acceptor3 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=103, name=qtp1442633686-103 Acceptor0 SelectChannelConnector@0.0.0.0:17211, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=28, name=qtp806943770-28 Acceptor1 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=29, name=qtp806943770-29 Acceptor2 SelectChannelConnector@0.0.0.0:19540, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest: 
   1) Thread[id=27, name=qtp806943770-27 Acceptor0 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=30, name=qtp806943770-30 Acceptor3 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=104, name=qtp1442633686-104 Acceptor1 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=105, name=qtp1442633686-105 Acceptor2 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=106, name=qtp1442633686-106 Acceptor3 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=103, name=qtp1442633686-103 Acceptor0 SelectChannelConnector@0.0.0.0:17211, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=28, name=qtp806943770-28 Acceptor1 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=29, name=qtp806943770-29 Acceptor2 SelectChannelConnector@0.0.0.0:19540, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([8573C2ED46DC4A5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=27, name=qtp806943770-27 Acceptor0 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=30, name=qtp806943770-30 Acceptor3 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=104, name=qtp1442633686-104 Acceptor1 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=105, name=qtp1442633686-105 Acceptor2 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=106, name=qtp1442633686-106 Acceptor3 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=103, name=qtp1442633686-103 Acceptor0 SelectChannelConnector@0.0.0.0:17211, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=28, name=qtp806943770-28 Acceptor1 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=29, name=qtp806943770-29 Acceptor2 SelectChannelConnector@0.0.0.0:19540, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=27, name=qtp806943770-27 Acceptor0 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=30, name=qtp806943770-30 Acceptor3 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=104, name=qtp1442633686-104 Acceptor1 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=105, name=qtp1442633686-105 Acceptor2 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=106, name=qtp1442633686-106 Acceptor3 SelectChannelConnector@0.0.0.0:17211, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=103, name=qtp1442633686-103 Acceptor0 SelectChannelConnector@0.0.0.0:17211, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=28, name=qtp806943770-28 Acceptor1 SelectChannelConnector@0.0.0.0:19540, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=29, name=qtp806943770-29 Acceptor2 SelectChannelConnector@0.0.0.0:19540, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([8573C2ED46DC4A5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedSpellCheckComponentTest

Error Message:
36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedSpellCheckComponentTest:     1) Thread[id=188, name=qtp109855678-188 Acceptor1 SelectChannelConnector@0.0.0.0:19714, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=153, name=qtp1942734199-153 Acceptor1 SelectChannelConnector@0.0.0.0:10507, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=142, name=qtp1432551238-142 Acceptor3 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=174, name=qtp1979690111-174 Acceptor1 SelectChannelConnector@0.0.0.0:15460, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=202, name=qtp406322006-202 Acceptor1 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=155, name=qtp1942734199-155 Acceptor3 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=204, name=qtp406322006-204 Acceptor3 SelectChannelConnector@0.0.0.0:30562, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=215, name=qtp453856425-215 Acceptor0 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=97, name=qtp1431523121-97 Acceptor1 SelectChannelConnector@0.0.0.0:19597, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=99, name=qtp1431523121-99 Acceptor3 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=187, name=qtp109855678-187 Acceptor0 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=216, name=qtp453856425-216 Acceptor1 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   13) Thread[id=129, name=qtp390636756-129 Acceptor3 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   14) Thread[id=112, name=qtp1507644908-112 Acceptor2 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   15) Thread[id=201, name=qtp406322006-201 Acceptor0 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   16) Thread[id=176, name=qtp1979690111-176 Acceptor3 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   17) Thread[id=98, name=qtp1431523121-98 Acceptor2 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   18) Thread[id=175, name=qtp1979690111-175 Acceptor2 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   19) Thread[id=126, name=qtp390636756-126 Acceptor0 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   20) Thread[id=113, name=qtp1507644908-113 Acceptor3 SelectChannelConnector@0.0.0.0:28779, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   21) Thread[id=190, name=qtp109855678-190 Acceptor3 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   22) Thread[id=111, name=qtp1507644908-111 Acceptor1 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   23) Thread[id=189, name=qtp109855678-189 Acceptor2 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   24) Thread[id=217, name=qtp453856425-217 Acceptor2 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   25) Thread[id=173, name=qtp1979690111-173 Acceptor0 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   26) Thread[id=152, name=qtp1942734199-152 Acceptor0 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   27) Thread[id=203, name=qtp406322006-203 Acceptor2 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   28) Thread[id=127, name=qtp390636756-127 Acceptor1 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   29) Thread[id=110, name=qtp1507644908-110 Acceptor0 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   30) Thread[id=154, name=qtp1942734199-154 Acceptor2 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   31) Thread[id=96, name=qtp1431523121-96 Acceptor0 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   32) Thread[id=140, name=qtp1432551238-140 Acceptor1 SelectChannelConnector@0.0.0.0:49279, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   33) Thread[id=218, name=qtp453856425-218 Acceptor3 SelectChannelConnector@0.0.0.0:25011, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   34) Thread[id=128, name=qtp390636756-128 Acceptor2 SelectChannelConnector@0.0.0.0:51378, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   35) Thread[id=141, name=qtp1432551238-141 Acceptor2 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   36) Thread[id=139, name=qtp1432551238-139 Acceptor0 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedSpellCheckComponentTest: 
   1) Thread[id=188, name=qtp109855678-188 Acceptor1 SelectChannelConnector@0.0.0.0:19714, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=153, name=qtp1942734199-153 Acceptor1 SelectChannelConnector@0.0.0.0:10507, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=142, name=qtp1432551238-142 Acceptor3 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=174, name=qtp1979690111-174 Acceptor1 SelectChannelConnector@0.0.0.0:15460, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=202, name=qtp406322006-202 Acceptor1 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=155, name=qtp1942734199-155 Acceptor3 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=204, name=qtp406322006-204 Acceptor3 SelectChannelConnector@0.0.0.0:30562, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=215, name=qtp453856425-215 Acceptor0 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=97, name=qtp1431523121-97 Acceptor1 SelectChannelConnector@0.0.0.0:19597, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=99, name=qtp1431523121-99 Acceptor3 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=187, name=qtp109855678-187 Acceptor0 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=216, name=qtp453856425-216 Acceptor1 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  13) Thread[id=129, name=qtp390636756-129 Acceptor3 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  14) Thread[id=112, name=qtp1507644908-112 Acceptor2 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  15) Thread[id=201, name=qtp406322006-201 Acceptor0 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  16) Thread[id=176, name=qtp1979690111-176 Acceptor3 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  17) Thread[id=98, name=qtp1431523121-98 Acceptor2 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  18) Thread[id=175, name=qtp1979690111-175 Acceptor2 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  19) Thread[id=126, name=qtp390636756-126 Acceptor0 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  20) Thread[id=113, name=qtp1507644908-113 Acceptor3 SelectChannelConnector@0.0.0.0:28779, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  21) Thread[id=190, name=qtp109855678-190 Acceptor3 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  22) Thread[id=111, name=qtp1507644908-111 Acceptor1 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  23) Thread[id=189, name=qtp109855678-189 Acceptor2 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  24) Thread[id=217, name=qtp453856425-217 Acceptor2 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  25) Thread[id=173, name=qtp1979690111-173 Acceptor0 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  26) Thread[id=152, name=qtp1942734199-152 Acceptor0 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  27) Thread[id=203, name=qtp406322006-203 Acceptor2 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  28) Thread[id=127, name=qtp390636756-127 Acceptor1 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  29) Thread[id=110, name=qtp1507644908-110 Acceptor0 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  30) Thread[id=154, name=qtp1942734199-154 Acceptor2 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  31) Thread[id=96, name=qtp1431523121-96 Acceptor0 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  32) Thread[id=140, name=qtp1432551238-140 Acceptor1 SelectChannelConnector@0.0.0.0:49279, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  33) Thread[id=218, name=qtp453856425-218 Acceptor3 SelectChannelConnector@0.0.0.0:25011, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  34) Thread[id=128, name=qtp390636756-128 Acceptor2 SelectChannelConnector@0.0.0.0:51378, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  35) Thread[id=141, name=qtp1432551238-141 Acceptor2 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  36) Thread[id=139, name=qtp1432551238-139 Acceptor0 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([8573C2ED46DC4A5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedSpellCheckComponentTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=188, name=qtp109855678-188 Acceptor1 SelectChannelConnector@0.0.0.0:19714, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    2) Thread[id=153, name=qtp1942734199-153 Acceptor1 SelectChannelConnector@0.0.0.0:10507, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    3) Thread[id=142, name=qtp1432551238-142 Acceptor3 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    4) Thread[id=174, name=qtp1979690111-174 Acceptor1 SelectChannelConnector@0.0.0.0:15460, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    5) Thread[id=202, name=qtp406322006-202 Acceptor1 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    6) Thread[id=155, name=qtp1942734199-155 Acceptor3 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    7) Thread[id=204, name=qtp406322006-204 Acceptor3 SelectChannelConnector@0.0.0.0:30562, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    8) Thread[id=215, name=qtp453856425-215 Acceptor0 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)    9) Thread[id=97, name=qtp1431523121-97 Acceptor1 SelectChannelConnector@0.0.0.0:19597, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   10) Thread[id=99, name=qtp1431523121-99 Acceptor3 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   11) Thread[id=187, name=qtp109855678-187 Acceptor0 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   12) Thread[id=216, name=qtp453856425-216 Acceptor1 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   13) Thread[id=129, name=qtp390636756-129 Acceptor3 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   14) Thread[id=112, name=qtp1507644908-112 Acceptor2 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   15) Thread[id=176, name=qtp1979690111-176 Acceptor3 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   16) Thread[id=201, name=qtp406322006-201 Acceptor0 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   17) Thread[id=98, name=qtp1431523121-98 Acceptor2 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   18) Thread[id=175, name=qtp1979690111-175 Acceptor2 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   19) Thread[id=113, name=qtp1507644908-113 Acceptor3 SelectChannelConnector@0.0.0.0:28779, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   20) Thread[id=126, name=qtp390636756-126 Acceptor0 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   21) Thread[id=111, name=qtp1507644908-111 Acceptor1 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   22) Thread[id=190, name=qtp109855678-190 Acceptor3 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   23) Thread[id=189, name=qtp109855678-189 Acceptor2 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   24) Thread[id=217, name=qtp453856425-217 Acceptor2 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   25) Thread[id=173, name=qtp1979690111-173 Acceptor0 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   26) Thread[id=152, name=qtp1942734199-152 Acceptor0 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   27) Thread[id=203, name=qtp406322006-203 Acceptor2 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   28) Thread[id=127, name=qtp390636756-127 Acceptor1 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   29) Thread[id=110, name=qtp1507644908-110 Acceptor0 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   30) Thread[id=154, name=qtp1942734199-154 Acceptor2 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   31) Thread[id=96, name=qtp1431523121-96 Acceptor0 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   32) Thread[id=140, name=qtp1432551238-140 Acceptor1 SelectChannelConnector@0.0.0.0:49279, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   33) Thread[id=218, name=qtp453856425-218 Acceptor3 SelectChannelConnector@0.0.0.0:25011, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   34) Thread[id=128, name=qtp390636756-128 Acceptor2 SelectChannelConnector@0.0.0.0:51378, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   35) Thread[id=141, name=qtp1432551238-141 Acceptor2 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)   36) Thread[id=139, name=qtp1432551238-139 Acceptor0 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=188, name=qtp109855678-188 Acceptor1 SelectChannelConnector@0.0.0.0:19714, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   2) Thread[id=153, name=qtp1942734199-153 Acceptor1 SelectChannelConnector@0.0.0.0:10507, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   3) Thread[id=142, name=qtp1432551238-142 Acceptor3 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   4) Thread[id=174, name=qtp1979690111-174 Acceptor1 SelectChannelConnector@0.0.0.0:15460, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   5) Thread[id=202, name=qtp406322006-202 Acceptor1 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   6) Thread[id=155, name=qtp1942734199-155 Acceptor3 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   7) Thread[id=204, name=qtp406322006-204 Acceptor3 SelectChannelConnector@0.0.0.0:30562, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   8) Thread[id=215, name=qtp453856425-215 Acceptor0 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
   9) Thread[id=97, name=qtp1431523121-97 Acceptor1 SelectChannelConnector@0.0.0.0:19597, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  10) Thread[id=99, name=qtp1431523121-99 Acceptor3 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  11) Thread[id=187, name=qtp109855678-187 Acceptor0 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  12) Thread[id=216, name=qtp453856425-216 Acceptor1 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  13) Thread[id=129, name=qtp390636756-129 Acceptor3 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  14) Thread[id=112, name=qtp1507644908-112 Acceptor2 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  15) Thread[id=176, name=qtp1979690111-176 Acceptor3 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  16) Thread[id=201, name=qtp406322006-201 Acceptor0 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  17) Thread[id=98, name=qtp1431523121-98 Acceptor2 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  18) Thread[id=175, name=qtp1979690111-175 Acceptor2 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  19) Thread[id=113, name=qtp1507644908-113 Acceptor3 SelectChannelConnector@0.0.0.0:28779, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  20) Thread[id=126, name=qtp390636756-126 Acceptor0 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  21) Thread[id=111, name=qtp1507644908-111 Acceptor1 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  22) Thread[id=190, name=qtp109855678-190 Acceptor3 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  23) Thread[id=189, name=qtp109855678-189 Acceptor2 SelectChannelConnector@0.0.0.0:19714, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  24) Thread[id=217, name=qtp453856425-217 Acceptor2 SelectChannelConnector@0.0.0.0:25011, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  25) Thread[id=173, name=qtp1979690111-173 Acceptor0 SelectChannelConnector@0.0.0.0:15460, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  26) Thread[id=152, name=qtp1942734199-152 Acceptor0 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  27) Thread[id=203, name=qtp406322006-203 Acceptor2 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  28) Thread[id=127, name=qtp390636756-127 Acceptor1 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  29) Thread[id=110, name=qtp1507644908-110 Acceptor0 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  30) Thread[id=154, name=qtp1942734199-154 Acceptor2 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  31) Thread[id=96, name=qtp1431523121-96 Acceptor0 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  32) Thread[id=140, name=qtp1432551238-140 Acceptor1 SelectChannelConnector@0.0.0.0:49279, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  33) Thread[id=218, name=qtp453856425-218 Acceptor3 SelectChannelConnector@0.0.0.0:25011, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  34) Thread[id=128, name=qtp390636756-128 Acceptor2 SelectChannelConnector@0.0.0.0:51378, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  35) Thread[id=141, name=qtp1432551238-141 Acceptor2 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
  36) Thread[id=139, name=qtp1432551238-139 Acceptor0 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([8573C2ED46DC4A5E]:0)




Build Log:
[...truncated 8346 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4]   2> 341 T11 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-RecoveryZkTest-1351959534976
[junit4:junit4]   2> 358 T11 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 461 T12 oaz.Environment.logEnv Server environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 462 T12 oaz.Environment.logEnv Server environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 463 T12 oaz.Environment.logEnv Server environment:java.version=1.7.0_09
[junit4:junit4]   2> 463 T12 oaz.Environment.logEnv Server environment:java.vendor=Oracle Corporation
[junit4:junit4]   2> 464 T12 oaz.Environment.logEnv Server environment:java.home=/usr/local/openjdk7/jre
[junit4:junit4]   2> 465 T12 oaz.Environment.logEnv Server environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk7/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 470 T12 oaz.Environment.logEnv Server environment:java.library.path=/usr/local/openjdk7/jre/lib/amd64/server:/usr/local/openjdk7/jre/lib/amd64:/usr/local/openjdk7/jre/../lib/amd64:/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 470 T12 oaz.Environment.logEnv Server environment:java.io.tmpdir=.
[junit4:junit4]   2> 471 T12 oaz.Environment.logEnv Server environment:java.compiler=<NA>
[junit4:junit4]   2> 472 T12 oaz.Environment.logEnv Server environment:os.name=FreeBSD
[junit4:junit4]   2> 472 T12 oaz.Environment.logEnv Server environment:os.arch=amd64
[junit4:junit4]   2> 473 T12 oaz.Environment.logEnv Server environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 473 T12 oaz.Environment.logEnv Server environment:user.name=hudson
[junit4:junit4]   2> 474 T12 oaz.Environment.logEnv Server environment:user.home=/home/hudson
[junit4:junit4]   2> 474 T12 oaz.Environment.logEnv Server environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 518 T12 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 556 T12 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 568 T11 oasc.ZkTestServer.run start zk server on port:63015
[junit4:junit4]   2> 641 T12 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 655 T11 oaz.Environment.logEnv Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 655 T11 oaz.Environment.logEnv Client environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 656 T11 oaz.Environment.logEnv Client environment:java.version=1.7.0_09
[junit4:junit4]   2> 657 T11 oaz.Environment.logEnv Client environment:java.vendor=Oracle Corporation
[junit4:junit4]   2> 657 T11 oaz.Environment.logEnv Client environment:java.home=/usr/local/openjdk7/jre
[junit4:junit4]   2> 658 T11 oaz.Environment.logEnv Client environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk7/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 661 T11 oaz.Environment.logEnv Client environment:java.library.path=/usr/local/openjdk7/jre/lib/amd64/server:/usr/local/openjdk7/jre/lib/amd64:/usr/local/openjdk7/jre/../lib/amd64:/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 661 T11 oaz.Environment.logEnv Client environment:java.io.tmpdir=.
[junit4:junit4]   2> 662 T11 oaz.Environment.logEnv Client environment:java.compiler=<NA>
[junit4:junit4]   2> 663 T11 oaz.Environment.logEnv Client environment:os.name=FreeBSD
[junit4:junit4]   2> 663 T11 oaz.Environment.logEnv Client environment:os.arch=amd64
[junit4:junit4]   2> 664 T11 oaz.Environment.logEnv Client environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 665 T11 oaz.Environment.logEnv Client environment:user.name=hudson
[junit4:junit4]   2> 666 T11 oaz.Environment.logEnv Client environment:user.home=/home/hudson
[junit4:junit4]   2> 666 T11 oaz.Environment.logEnv Client environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 671 T11 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63015 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@790d63f8
[junit4:junit4]   2> 731 T15 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63015
[junit4:junit4]   2> 731 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 736 T15 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63015, initiating session
[junit4:junit4]   2> 737 T13 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:30574
[junit4:junit4]   2> 760 T13 oazs.NIOServerCnxn.doIO WARNING Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
[junit4:junit4]   2> 761 T13 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:30574 (no session established for client)
[junit4:junit4]   2> 762 T15 oaz.ClientCnxn$SendThread.run Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
[junit4:junit4]   2> 2187 T15 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:63015
[junit4:junit4]   2> 2188 T15 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63015, initiating session
[junit4:junit4]   2> 2189 T13 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:26698
[junit4:junit4]   2> 2191 T13 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:26698
[junit4:junit4]   2> 2208 T16 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 2245 T16 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac7115ecc0000 with negotiated timeout 10000 for client /140.211.11.196:26698
[junit4:junit4]   2> 2246 T15 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63015, sessionid = 0x13ac7115ecc0000, negotiated timeout = 10000
[junit4:junit4]   2> 2278 T17 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@790d63f8 name:ZooKeeperConnection Watcher:127.0.0.1:63015 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2279 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2285 T11 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 2389 T18 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac7115ecc0000
[junit4:junit4]   2> 2392 T17 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2391 T11 oaz.ZooKeeper.close Session: 0x13ac7115ecc0000 closed
[junit4:junit4]   2> 2393 T13 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:26698 which had sessionid 0x13ac7115ecc0000
[junit4:junit4]   2> 2393 T11 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63015/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@5702cbd3
[junit4:junit4]   2> 2396 T19 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63015
[junit4:junit4]   2> 2396 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2397 T19 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63015, initiating session
[junit4:junit4]   2> 2398 T13 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:10099
[junit4:junit4]   2> 2399 T13 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:10099
[junit4:junit4]   2> 2403 T16 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac7115ecc0001 with negotiated timeout 10000 for client /140.211.11.196:10099
[junit4:junit4]   2> 2403 T19 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63015, sessionid = 0x13ac7115ecc0001, negotiated timeout = 10000
[junit4:junit4]   2> 2404 T20 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5702cbd3 name:ZooKeeperConnection Watcher:127.0.0.1:63015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2405 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2427 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 2435 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 2441 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 2446 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 2452 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2523 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2540 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 2542 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 2549 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2550 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 2572 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2573 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 2579 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 2580 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 2585 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 2586 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 2591 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2592 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 2597 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2599 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 2604 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2605 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 2610 T11 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2611 T11 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 2616 T18 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac7115ecc0001
[junit4:junit4]   2> 2618 T20 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2618 T11 oaz.ZooKeeper.close Session: 0x13ac7115ecc0001 closed
[junit4:junit4]   2> 2619 T13 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:10099 which had sessionid 0x13ac7115ecc0001
[junit4:junit4]   2> 3083 T11 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3380 T11 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:19540
[junit4:junit4]   2> 3667 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3669 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3670 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 4662 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4668 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4668 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 4713 T11 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 4719 T11 oasc.CoreContainer.<init> New CoreContainer 1069263533
[junit4:junit4]   2> 4720 T11 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 4721 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 4829 T11 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4864 T11 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:63015/solr
[junit4:junit4]   2> 4920 T11 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63015/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@7659657a
[junit4:junit4]   2> 4921 T35 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63015
[junit4:junit4]   2> 4921 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4923 T35 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63015, initiating session
[junit4:junit4]   2> 4924 T13 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:31390
[junit4:junit4]   2> 4926 T13 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:31390
[junit4:junit4]   2> 4929 T16 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac7115ecc0002 with negotiated timeout 5000 for client /140.211.11.196:31390
[junit4:junit4]   2> 4929 T35 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63015, sessionid = 0x13ac7115ecc0002, negotiated timeout = 5000
[junit4:junit4]   2> 4930 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7659657a name:ZooKeeperConnection Watcher:127.0.0.1:63015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4930 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4953 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4966 T11 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 4970 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:19540_solr
[junit4:junit4]   2> 4977 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:19540_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:19540_solr
[junit4:junit4]   2> 4978 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:19540_solr
[junit4:junit4]   2> 5004 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 5005 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 5005 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 5006 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 5006 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 5007 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 5008 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 5008 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 5009 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 5081 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 5723 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 5762 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 5764 T11 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 5768 T11 oasc.Overseer.start Overseer (id=88602020105945090-127.0.0.1:19540_solr-n_0000000000) starting
[junit4:junit4]   2> 5769 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5785 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5787 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5793 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5795 T38 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 5796 T11 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 5800 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 5824 T37 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 5889 T11 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 5890 T11 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 5891 T11 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5892 T11 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 5894 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 5896 T11 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5898 T11 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 6135 T11 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6395 T11 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 6425 T11 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6500 T11 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 8633 T11 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 8668 T11 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 8673 T11 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 9058 T11 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9081 T11 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9088 T11 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9097 T11 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9189 T11 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351959534975/control/data/
[junit4:junit4]   2> 9196 T11 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@12f794ce
[junit4:junit4]   2> 9458 T11 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351959534975/control/data/index/
[junit4:junit4]   2> 9460 T11 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351959534975/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 9573 T11 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351959534975/control/data/index forceNew:false
[junit4:junit4]   2> 10084 T11 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351959534975/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@46f0474a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10085 T11 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10118 T11 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 11080 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 11082 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 11082 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 11092 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 11232 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 11247 T11 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 11253 T11 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 11290 T11 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 11310 T11 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 11359 T11 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 11361 T11 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 11367 T11 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 11368 T11 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 11369 T11 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 11369 T11 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 11398 T11 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 11399 T11 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 11400 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 11401 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 11403 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 11404 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 11405 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 11406 T11 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 11426 T11 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 11445 T11 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 11574 T11 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 11608 T11 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 11617 T11 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 11677 T11 oass.SolrIndexSearcher.<init> Opening Searcher@38219f32 main
[junit4:junit4]   2> 11765 T11 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 11765 T11 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 11767 T11 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 11873 T11 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 12273 T39 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38219f32 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 12291 T11 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 12293 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x74 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 12306 T11 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 12307 T11 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:19540/solr shardId:control_shard
[junit4:junit4]   2> 12324 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 12341 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 12359 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:delete cxid:0x84 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 12366 T11 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 12368 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x88 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12368 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12370 T37 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:19540_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:19540/solr"}
[junit4:junit4]   2> 12386 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0x8b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 12426 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 12873 T11 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 12874 T11 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 12875 T11 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:19540/solr/collection1/
[junit4:junit4]   2> 12876 T11 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 12876 T11 oasc.SyncStrategy.syncToMe http://127.0.0.1:19540/solr/collection1/ has no replicas
[junit4:junit4]   2> 12877 T11 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:19540/solr/collection1/
[junit4:junit4]   2> 12878 T11 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 12885 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0002 type:create cxid:0xa9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12935 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12949 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 12998 T11 oasc.ZkController.register We are http://127.0.0.1:19540/solr/collection1/ and leader is http://127.0.0.1:19540/solr/collection1/
[junit4:junit4]   2> 13003 T11 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:19540/solr
[junit4:junit4]   2> 13003 T11 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 13004 T11 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 13009 T11 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13010 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1
[junit4:junit4]   2> 13011 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 13088 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13303 T11 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 13306 T11 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:34199
[junit4:junit4]   2> 13308 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13308 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740
[junit4:junit4]   2> 13309 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740/'
[junit4:junit4]   2> 13360 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 13361 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13361 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740
[junit4:junit4]   2> 13362 T11 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740/solr.xml
[junit4:junit4]   2> 13363 T11 oasc.CoreContainer.<init> New CoreContainer 1693106741
[junit4:junit4]   2> 13364 T11 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740/'
[junit4:junit4]   2> 13365 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740/'
[junit4:junit4]   2> 13407 T11 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13429 T11 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:63015/solr
[junit4:junit4]   2> 13430 T11 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63015/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@768db0c6
[junit4:junit4]   2> 13432 T49 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63015
[junit4:junit4]   2> 13432 T11 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13434 T49 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63015, initiating session
[junit4:junit4]   2> 13434 T13 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:57041
[junit4:junit4]   2> 13435 T13 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:57041
[junit4:junit4]   2> 13518 T16 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac7115ecc0003 with negotiated timeout 5000 for client /140.211.11.196:57041
[junit4:junit4]   2> 13518 T49 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63015, sessionid = 0x13ac7115ecc0003, negotiated timeout = 5000
[junit4:junit4]   2> 13520 T50 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@768db0c6 name:ZooKeeperConnection Watcher:127.0.0.1:63015/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13520 T11 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13521 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13522 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13524 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13525 T37 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:19540_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:19540/solr"}
[junit4:junit4]   2> 13536 T11 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34199_solr
[junit4:junit4]   2> 13537 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:34199_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:34199_solr
[junit4:junit4]   2> 13539 T11 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34199_solr
[junit4:junit4]   2> 13542 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 13546 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 13547 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 13548 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 13548 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 13549 T36 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 13549 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 13550 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 13550 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 13551 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 13551 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 13552 T11 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 13577 T11 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13585 T11 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740/collection1
[junit4:junit4]   2> 13586 T11 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 13587 T11 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 13588 T11 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 13590 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740/collection1/'
[junit4:junit4]   2> 13592 T11 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty1-1351959547740/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 13593 T11 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/solrtest-jetty1-1351959547740/collection1/lib/README' to classloader
[junit4:junit4]   2> 13687 T11 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 13774 T11 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 13776 T11 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 13822 T11 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 14489 T11 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 14504 T11 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 14509 T11 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 14537 T11 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14544 T11 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14550 T11 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14554 T11 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14559 T11 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./solrtest-jetty1-1351959547740/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351959534975/jetty1/
[junit4:junit4]   2> 14559 T11 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@12f794ce
[junit4:junit4]   2> 14561 T11 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351959534975/jetty1/index/
[junit4:junit4]   2> 14562 T11 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351959534975/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 14563 T11 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351959534975/jetty1/index forceNew:false
[junit4:junit4]   2> 14571 T11 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351959534975/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@82d195a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 14572 T11 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 14574 T11 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 14579 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 14580 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 14581 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 14582 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 14583 T11 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 14584 T11 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 14585 T11 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 14587 T11 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 14589 T11 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 14590 T11 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 14592 T11 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 14593 T11 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 14594 T11 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 14595 T11 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 14596 T11 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 14597 T11 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 14599 T11 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 14600 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 14602 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 14603 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 14605 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 14606 T11 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 14609 T11 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 14612 T11 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 14617 T11 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 14637 T11 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 14642 T11 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 14644 T11 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 14648 T11 oass.SolrIndexSearcher.<init> Opening Searcher@6b861ad2 main
[junit4:junit4]   2> 14650 T11 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 14650 T11 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 14652 T11 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 14682 T11 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 14853 T55 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b861ad2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 14865 T11 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15054 T37 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15055 T37 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:34199_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:34199/solr"}
[junit4:junit4]   2> 15055 T37 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 15084 T50 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 15084 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 15389 T11 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 15389 T11 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:34199/solr shardId:shard1
[junit4:junit4]   2> 15391 T11 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 15411 T18 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac7115ecc0003 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr

[...truncated too long message...]

se.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   27) Thread[id=203, name=qtp406322006-203 Acceptor2 SelectChannelConnector@0.0.0.0:30562, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   28) Thread[id=127, name=qtp390636756-127 Acceptor1 SelectChannelConnector@0.0.0.0:51378, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   29) Thread[id=110, name=qtp1507644908-110 Acceptor0 SelectChannelConnector@0.0.0.0:28779, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   30) Thread[id=154, name=qtp1942734199-154 Acceptor2 SelectChannelConnector@0.0.0.0:10507, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   31) Thread[id=96, name=qtp1431523121-96 Acceptor0 SelectChannelConnector@0.0.0.0:19597, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   32) Thread[id=140, name=qtp1432551238-140 Acceptor1 SelectChannelConnector@0.0.0.0:49279, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   33) Thread[id=218, name=qtp453856425-218 Acceptor3 SelectChannelConnector@0.0.0.0:25011, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   34) Thread[id=128, name=qtp390636756-128 Acceptor2 SelectChannelConnector@0.0.0.0:51378, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   35) Thread[id=141, name=qtp1432551238-141 Acceptor2 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    >   36) Thread[id=139, name=qtp1432551238-139 Acceptor0 SelectChannelConnector@0.0.0.0:49279, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([8573C2ED46DC4A5E]:0)
[junit4:junit4] Completed on J0 in 40.15s, 1 test, 2 errors <<< FAILURES!

[...truncated 473 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-java7/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 825 ignored

Total time: 31 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message