Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D3B4AD3E5 for ; Sat, 27 Oct 2012 07:40:29 +0000 (UTC) Received: (qmail 24907 invoked by uid 500); 27 Oct 2012 07:40:28 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 24709 invoked by uid 500); 27 Oct 2012 07:40:27 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 24685 invoked by uid 99); 27 Oct 2012 07:40:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Oct 2012 07:40:26 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,NORMAL_HTTP_TO_IP,WEIRD_PORT X-Spam-Check-By: apache.org Received: from [140.211.11.8] (HELO aegis.apache.org) (140.211.11.8) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Oct 2012 07:40:17 +0000 Received: from aegis.apache.org (localhost [127.0.0.1]) by aegis.apache.org (Postfix) with ESMTP id 756BBC00A0 for ; Sat, 27 Oct 2012 07:39:57 +0000 (UTC) Date: Sat, 27 Oct 2012 07:39:42 +0000 (UTC) From: Apache Jenkins Server To: dev@lucene.apache.org Message-ID: <142694471.560.1351323597465.JavaMail.hudson@aegis> In-Reply-To: <547725787.310.1351244757530.JavaMail.hudson@aegis> References: <547725787.310.1351244757530.JavaMail.hudson@aegis> Subject: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 74 - Still Failing MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_559_14101358.1351323582978" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_559_14101358.1351323582978 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/74/ 4 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest Error Message: 12 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:= 1) Thread[id=3D139, name=3Dqtp2080562400-139 Acceptor0 SelectChannelCo= nnector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java= :147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept= (SelectChannelConnector.java:104) at org.eclipse.jetty.server.Abstr= actConnector$Acceptor.run(AbstractConnector.java:933) at org.eclips= e.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPoo= l.java:534) at java.lang.Thread.run(Thread.java:679) 2) Thread[i= d=3D142, name=3Dqtp2080562400-142 Acceptor3 SelectChannelConnector@0.0.0.0:= 51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.= ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) at= org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelCo= nnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acc= eptor.run(AbstractConnector.java:933) at org.eclipse.jetty.util.thr= ead.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclip= se.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) = at java.lang.Thread.run(Thread.java:679) 3) Thread[id=3D141, name=3D= qtp2080562400-141 Acceptor2 SelectChannelConnector@0.0.0.0:51150, state=3DB= LOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChan= nelImpl.accept(ServerSocketChannelImpl.java:147) at org.eclipse.jet= ty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104= ) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstra= ctConnector.java:933) at org.eclipse.jetty.util.thread.QueuedThread= Pool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.th= read.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang= .Thread.run(Thread.java:679) 4) Thread[id=3D121, name=3Dqtp190034072-121= Acceptor2 SelectChannelConnector@0.0.0.0:60207, state=3DBLOCKED, group=3DT= GRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(Se= rverSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio.Sel= ectChannelConnector.accept(SelectChannelConnector.java:104) at org.= eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:= 933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThread= Pool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Threa= d.java:679) 5) Thread[id=3D119, name=3Dqtp190034072-119 Acceptor0 Select= ChannelConnector@0.0.0.0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTes= t] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannel= Impl.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnect= or.accept(SelectChannelConnector.java:104) at org.eclipse.jetty.ser= ver.AbstractConnector$Acceptor.run(AbstractConnector.java:933) at o= rg.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:= 599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(Queued= ThreadPool.java:534) at java.lang.Thread.run(Thread.java:679) 6)= Thread[id=3D294, name=3Dqtp2136594195-294 Acceptor2 SelectChannelConnector= @0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at su= n.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Select= ChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConn= ector$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty= .util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at = org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:= 534) at java.lang.Thread.run(Thread.java:679) 7) Thread[id=3D140= , name=3Dqtp2080562400-140 Acceptor1 SelectChannelConnector@0.0.0.0:51150, = state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.Server= SocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocket= ChannelImpl.accept(ServerSocketChannelImpl.java:163) at org.eclipse= .jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java= :104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Ab= stractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedTh= readPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.uti= l.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.= lang.Thread.run(Thread.java:679) 8) Thread[id=3D122, name=3Dqtp190034072= -122 Acceptor3 SelectChannelConnector@0.0.0.0:60207, state=3DBLOCKED, group= =3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accep= t(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio= .SelectChannelConnector.accept(SelectChannelConnector.java:104) at = org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.j= ava:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Q= ueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedTh= readPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(T= hread.java:679) 9) Thread[id=3D120, name=3Dqtp190034072-120 Acceptor1 Se= lectChannelConnector@0.0.0.0:60207, state=3DRUNNABLE, group=3DTGRP-Recovery= ZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method= ) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelI= mpl.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnecto= r.accept(SelectChannelConnector.java:104) at org.eclipse.jetty.serv= er.AbstractConnector$Acceptor.run(AbstractConnector.java:933) at or= g.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:5= 99) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT= hreadPool.java:534) at java.lang.Thread.run(Thread.java:679) 10) = Thread[id=3D293, name=3Dqtp2136594195-293 Acceptor1 SelectChannelConnector@= 0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun= .nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectC= hannelConnector.java:104) at org.eclipse.jetty.server.AbstractConne= ctor$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.= util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at o= rg.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:5= 34) at java.lang.Thread.run(Thread.java:679) 11) Thread[id=3D292,= name=3Dqtp2136594195-292 Acceptor0 SelectChannelConnector@0.0.0.0:27739, s= tate=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerS= ocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketC= hannelImpl.accept(ServerSocketChannelImpl.java:163) at org.eclipse.= jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:= 104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abs= tractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedThr= eadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util= .thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.l= ang.Thread.run(Thread.java:679) 12) Thread[id=3D295, name=3Dqtp2136594195= -295 Acceptor3 SelectChannelConnector@0.0.0.0:27739, state=3DBLOCKED, group= =3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accep= t(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio= .SelectChannelConnector.accept(SelectChannelConnector.java:104) at = org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.j= ava:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Q= ueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedTh= readPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(T= hread.java:679) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from = SUITE scope at org.apache.solr.cloud.RecoveryZkTest:=20 1) Thread[id=3D139, name=3Dqtp2080562400-139 Acceptor0 SelectChannelConn= ector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 2) Thread[id=3D142, name=3Dqtp2080562400-142 Acceptor3 SelectChannelConn= ector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 3) Thread[id=3D141, name=3Dqtp2080562400-141 Acceptor2 SelectChannelConn= ector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 4) Thread[id=3D121, name=3Dqtp190034072-121 Acceptor2 SelectChannelConne= ctor@0.0.0.0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 5) Thread[id=3D119, name=3Dqtp190034072-119 Acceptor0 SelectChannelConne= ctor@0.0.0.0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 6) Thread[id=3D294, name=3Dqtp2136594195-294 Acceptor2 SelectChannelConn= ector@0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 7) Thread[id=3D140, name=3Dqtp2080562400-140 Acceptor1 SelectChannelConn= ector@0.0.0.0:51150, state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 8) Thread[id=3D122, name=3Dqtp190034072-122 Acceptor3 SelectChannelConne= ctor@0.0.0.0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 9) Thread[id=3D120, name=3Dqtp190034072-120 Acceptor1 SelectChannelConne= ctor@0.0.0.0:60207, state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 10) Thread[id=3D293, name=3Dqtp2136594195-293 Acceptor1 SelectChannelConn= ector@0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 11) Thread[id=3D292, name=3Dqtp2136594195-292 Acceptor0 SelectChannelConn= ector@0.0.0.0:27739, state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 12) Thread[id=3D295, name=3Dqtp2136594195-295 Acceptor3 SelectChannelConn= ector@0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) =09at __randomizedtesting.SeedInfo.seed([A6F29E4E81D42F0D]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest Error Message: There are still zombie threads that couldn't be terminated: 1) Thread[id= =3D121, name=3Dqtp190034072-121 Acceptor2 SelectChannelConnector@0.0.0.0:60= 207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.Se= rverSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) at o= rg.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConn= ector.java:104) at org.eclipse.jetty.server.AbstractConnector$Accep= tor.run(AbstractConnector.java:933) at org.eclipse.jetty.util.threa= d.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse= .jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) = at java.lang.Thread.run(Thread.java:679) 2) Thread[id=3D119, name=3Dqt= p190034072-119 Acceptor0 SelectChannelConnector@0.0.0.0:60207, state=3DBLOC= KED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannel= Impl.accept(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.= server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) = at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractC= onnector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPoo= l.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.threa= d.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Th= read.run(Thread.java:679) 3) Thread[id=3D294, name=3Dqtp2136594195-294 A= cceptor2 SelectChannelConnector@0.0.0.0:27739, state=3DBLOCKED, group=3DTGR= P-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(Serv= erSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio.Selec= tChannelConnector.accept(SelectChannelConnector.java:104) at org.ec= lipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:93= 3) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedT= hreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPo= ol$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.= java:679) 4) Thread[id=3D140, name=3Dqtp2080562400-140 Acceptor1 SelectC= hannelConnector@0.0.0.0:51150, state=3DRUNNABLE, group=3DTGRP-RecoveryZkTes= t] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.j= ava:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.acc= ept(SelectChannelConnector.java:104) at org.eclipse.jetty.server.Ab= stractConnector$Acceptor.run(AbstractConnector.java:933) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThread= Pool.java:534) at java.lang.Thread.run(Thread.java:679) 5) Threa= d[id=3D122, name=3Dqtp190034072-122 Acceptor3 SelectChannelConnector@0.0.0.= 0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.c= h.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannel= Connector.java:104) at org.eclipse.jetty.server.AbstractConnector$A= cceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.util.t= hread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) = at java.lang.Thread.run(Thread.java:679) 6) Thread[id=3D120, name= =3Dqtp190034072-120 Acceptor1 SelectChannelConnector@0.0.0.0:60207, state= =3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocke= tChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChann= elImpl.accept(ServerSocketChannelImpl.java:163) at org.eclipse.jett= y.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)= at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstrac= tConnector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thr= ead.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.= Thread.run(Thread.java:679) 7) Thread[id=3D139, name=3Dqtp2080562400-139= Acceptor0 SelectChannelConnector@0.0.0.0:51150, state=3DBLOCKED, group=3DT= GRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(Se= rverSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio.Sel= ectChannelConnector.accept(SelectChannelConnector.java:104) at org.= eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:= 933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThread= Pool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Threa= d.java:679) 8) Thread[id=3D142, name=3Dqtp2080562400-142 Acceptor3 Selec= tChannelConnector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTe= st] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChanne= lImpl.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnec= tor.accept(SelectChannelConnector.java:104) at org.eclipse.jetty.se= rver.AbstractConnector$Acceptor.run(AbstractConnector.java:933) at = org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java= :599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(Queue= dThreadPool.java:534) at java.lang.Thread.run(Thread.java:679) 9= ) Thread[id=3D293, name=3Dqtp2136594195-293 Acceptor1 SelectChannelConnecto= r@0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at s= un.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractCon= nector$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jett= y.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at= org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java= :534) at java.lang.Thread.run(Thread.java:679) 10) Thread[id=3D29= 2, name=3Dqtp2136594195-292 Acceptor0 SelectChannelConnector@0.0.0.0:27739,= state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.Serve= rSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocke= tChannelImpl.accept(ServerSocketChannelImpl.java:163) at org.eclips= e.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.jav= a:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(A= bstractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedT= hreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java= .lang.Thread.run(Thread.java:679) 11) Thread[id=3D295, name=3Dqtp21365941= 95-295 Acceptor3 SelectChannelConnector@0.0.0.0:27739, state=3DBLOCKED, gro= up=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.acc= ept(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.server.n= io.SelectChannelConnector.accept(SelectChannelConnector.java:104) a= t org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector= .java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob= (QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.Queued= ThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run= (Thread.java:679) 12) Thread[id=3D141, name=3Dqtp2080562400-141 Acceptor2= SelectChannelConnector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-Recove= ryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocket= ChannelImpl.java:147) at org.eclipse.jetty.server.nio.SelectChannel= Connector.accept(SelectChannelConnector.java:104) at org.eclipse.je= tty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) = at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo= l.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run= (QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:679= ) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie = threads that couldn't be terminated: 1) Thread[id=3D121, name=3Dqtp190034072-121 Acceptor2 SelectChannelConne= ctor@0.0.0.0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 2) Thread[id=3D119, name=3Dqtp190034072-119 Acceptor0 SelectChannelConne= ctor@0.0.0.0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 3) Thread[id=3D294, name=3Dqtp2136594195-294 Acceptor2 SelectChannelConn= ector@0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 4) Thread[id=3D140, name=3Dqtp2080562400-140 Acceptor1 SelectChannelConn= ector@0.0.0.0:51150, state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 5) Thread[id=3D122, name=3Dqtp190034072-122 Acceptor3 SelectChannelConne= ctor@0.0.0.0:60207, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 6) Thread[id=3D120, name=3Dqtp190034072-120 Acceptor1 SelectChannelConne= ctor@0.0.0.0:60207, state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 7) Thread[id=3D139, name=3Dqtp2080562400-139 Acceptor0 SelectChannelConn= ector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 8) Thread[id=3D142, name=3Dqtp2080562400-142 Acceptor3 SelectChannelConn= ector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 9) Thread[id=3D293, name=3Dqtp2136594195-293 Acceptor1 SelectChannelConn= ector@0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 10) Thread[id=3D292, name=3Dqtp2136594195-292 Acceptor0 SelectChannelConn= ector@0.0.0.0:27739, state=3DRUNNABLE, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 11) Thread[id=3D295, name=3Dqtp2136594195-295 Acceptor3 SelectChannelConn= ector@0.0.0.0:27739, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 12) Thread[id=3D141, name=3Dqtp2080562400-141 Acceptor2 SelectChannelConn= ector@0.0.0.0:51150, state=3DBLOCKED, group=3DTGRP-RecoveryZkTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) =09at __randomizedtesting.SeedInfo.seed([A6F29E4E81D42F0D]:0) FAILED: junit.framework.TestSuite.org.apache.solr.update.SolrCmdDistributo= rTest Error Message: 20 threads leaked from SUITE scope at org.apache.solr.update.SolrCmdDistrib= utorTest: 1) Thread[id=3D244, name=3Dqtp149836331-244 Acceptor1 SelectC= hannelConnector@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistrib= utorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocket= ChannelImpl.java:147) at org.eclipse.jetty.server.nio.SelectChannel= Connector.accept(SelectChannelConnector.java:104) at org.eclipse.je= tty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) = at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo= l.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run= (QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:679= ) 2) Thread[id=3D232, name=3Dqtp837657752-232 Acceptor3 SelectChannelCon= nector@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest]= at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelIm= pl.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector= .accept(SelectChannelConnector.java:104) at org.eclipse.jetty.serve= r.AbstractConnector$Acceptor.run(AbstractConnector.java:933) at org= .eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:59= 9) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedTh= readPool.java:534) at java.lang.Thread.run(Thread.java:679) 3) T= hread[id=3D204, name=3Dqtp1236323694-204 Acceptor0 SelectChannelConnector@0= .0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:= 147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(= SelectChannelConnector.java:104) at org.eclipse.jetty.server.Abstra= ctConnector$Acceptor.run(AbstractConnector.java:933) at org.eclipse= .jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool= .java:534) at java.lang.Thread.run(Thread.java:679) 4) Thread[id= =3D216, name=3Dqtp144414042-216 Acceptor0 SelectChannelConnector@0.0.0.0:48= 493, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.n= io.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectCha= nnelConnector.java:104) at org.eclipse.jetty.server.AbstractConnect= or$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org= .eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534= ) at java.lang.Thread.run(Thread.java:679) 5) Thread[id=3D231, n= ame=3Dqtp837657752-231 Acceptor2 SelectChannelConnector@0.0.0.0:53656, stat= e=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.Ser= verSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) at or= g.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConne= ctor.java:104) at org.eclipse.jetty.server.AbstractConnector$Accept= or.run(AbstractConnector.java:933) at org.eclipse.jetty.util.thread= .QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.= jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) = at java.lang.Thread.run(Thread.java:679) 6) Thread[id=3D206, name=3Dqtp= 1236323694-206 Acceptor2 SelectChannelConnector@0.0.0.0:31983, state=3DBLOC= KED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocke= tChannelImpl.accept(ServerSocketChannelImpl.java:147) at org.eclips= e.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.jav= a:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(A= bstractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedT= hreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java= .lang.Thread.run(Thread.java:679) 7) Thread[id=3D205, name=3Dqtp12363236= 94-205 Acceptor1 SelectChannelConnector@0.0.0.0:31983, state=3DRUNNABLE, gr= oup=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChanne= lImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.= accept(ServerSocketChannelImpl.java:163) at org.eclipse.jetty.serve= r.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) = at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnec= tor.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.run= Job(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.Que= uedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.= run(Thread.java:679) 8) Thread[id=3D192, name=3Dqtp643552837-192 Accepto= r1 SelectChannelConnector@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-Solr= CmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(Se= rverSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio.Sel= ectChannelConnector.accept(SelectChannelConnector.java:104) at org.= eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:= 933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThread= Pool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Threa= d.java:679) 9) Thread[id=3D191, name=3Dqtp643552837-191 Acceptor0 Select= ChannelConnector@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistri= butorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocke= tChannelImpl.java:147) at org.eclipse.jetty.server.nio.SelectChanne= lConnector.accept(SelectChannelConnector.java:104) at org.eclipse.j= etty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) = at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPo= ol.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.ru= n(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:67= 9) 10) Thread[id=3D230, name=3Dqtp837657752-230 Acceptor1 SelectChannelCo= nnector@0.0.0.0:53656, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTes= t] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.j= ava:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.acc= ept(SelectChannelConnector.java:104) at org.eclipse.jetty.server.Ab= stractConnector$Acceptor.run(AbstractConnector.java:933) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThread= Pool.java:534) at java.lang.Thread.run(Thread.java:679) 11) Threa= d[id=3D193, name=3Dqtp643552837-193 Acceptor2 SelectChannelConnector@0.0.0.= 0:36458, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at = sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.ni= o.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChan= nelConnector.java:104) at org.eclipse.jetty.server.AbstractConnecto= r$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.uti= l.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.= eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)= at java.lang.Thread.run(Thread.java:679) 12) Thread[id=3D229, na= me=3Dqtp837657752-229 Acceptor0 SelectChannelConnector@0.0.0.0:53656, state= =3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.Serv= erSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) at org= .eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnec= tor.java:104) at org.eclipse.jetty.server.AbstractConnector$Accepto= r.run(AbstractConnector.java:933) at org.eclipse.jetty.util.thread.= QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.j= etty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) = at java.lang.Thread.run(Thread.java:679) 13) Thread[id=3D217, name=3Dqtp1= 44414042-217 Acceptor1 SelectChannelConnector@0.0.0.0:48493, state=3DBLOCKE= D, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketC= hannelImpl.accept(ServerSocketChannelImpl.java:147) at org.eclipse.= jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:= 104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abs= tractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedThr= eadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util= .thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.l= ang.Thread.run(Thread.java:679) 14) Thread[id=3D207, name=3Dqtp1236323694= -207 Acceptor3 SelectChannelConnector@0.0.0.0:31983, state=3DBLOCKED, group= =3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelIm= pl.accept(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.se= rver.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) = at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractCon= nector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.= runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.= QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thre= ad.run(Thread.java:679) 15) Thread[id=3D218, name=3Dqtp144414042-218 Acce= ptor2 SelectChannelConnector@0.0.0.0:48493, state=3DRUNNABLE, group=3DTGRP-= SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accep= t0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(Serv= erSocketChannelImpl.java:163) at org.eclipse.jetty.server.nio.Selec= tChannelConnector.accept(SelectChannelConnector.java:104) at org.ec= lipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:93= 3) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedT= hreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPo= ol$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.= java:679) 16) Thread[id=3D245, name=3Dqtp149836331-245 Acceptor2 SelectCh= annelConnector@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistribu= torTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketC= hannelImpl.java:147) at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) at org.eclipse.jet= ty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) = at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool= .java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(= QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:679)= 17) Thread[id=3D243, name=3Dqtp149836331-243 Acceptor0 SelectChannelConn= ector@0.0.0.0:16444, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest]= at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.jav= a:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accep= t(SelectChannelConnector.java:104) at org.eclipse.jetty.server.Abst= ractConnector$Acceptor.run(AbstractConnector.java:933) at org.eclip= se.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPo= ol.java:534) at java.lang.Thread.run(Thread.java:679) 18) Thread[= id=3D194, name=3Dqtp643552837-194 Acceptor3 SelectChannelConnector@0.0.0.0:= 36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun= .nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectC= hannelConnector.java:104) at org.eclipse.jetty.server.AbstractConne= ctor$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.= util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at o= rg.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:5= 34) at java.lang.Thread.run(Thread.java:679) 19) Thread[id=3D246,= name=3Dqtp149836331-246 Acceptor3 SelectChannelConnector@0.0.0.0:16444, st= ate=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.S= erverSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) at = org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelCon= nector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acce= ptor.run(AbstractConnector.java:933) at org.eclipse.jetty.util.thre= ad.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclips= e.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) = at java.lang.Thread.run(Thread.java:679) 20) Thread[id=3D219, name=3Dq= tp144414042-219 Acceptor3 SelectChannelConnector@0.0.0.0:48493, state=3DBLO= CKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSock= etChannelImpl.accept(ServerSocketChannelImpl.java:147) at org.eclip= se.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.ja= va:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(= AbstractConnector.java:933) at org.eclipse.jetty.util.thread.Queued= ThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.u= til.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at jav= a.lang.Thread.run(Thread.java:679) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from = SUITE scope at org.apache.solr.update.SolrCmdDistributorTest:=20 1) Thread[id=3D244, name=3Dqtp149836331-244 Acceptor1 SelectChannelConne= ctor@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 2) Thread[id=3D232, name=3Dqtp837657752-232 Acceptor3 SelectChannelConne= ctor@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 3) Thread[id=3D204, name=3Dqtp1236323694-204 Acceptor0 SelectChannelConn= ector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 4) Thread[id=3D216, name=3Dqtp144414042-216 Acceptor0 SelectChannelConne= ctor@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 5) Thread[id=3D231, name=3Dqtp837657752-231 Acceptor2 SelectChannelConne= ctor@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 6) Thread[id=3D206, name=3Dqtp1236323694-206 Acceptor2 SelectChannelConn= ector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 7) Thread[id=3D205, name=3Dqtp1236323694-205 Acceptor1 SelectChannelConn= ector@0.0.0.0:31983, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 8) Thread[id=3D192, name=3Dqtp643552837-192 Acceptor1 SelectChannelConne= ctor@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 9) Thread[id=3D191, name=3Dqtp643552837-191 Acceptor0 SelectChannelConne= ctor@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 10) Thread[id=3D230, name=3Dqtp837657752-230 Acceptor1 SelectChannelConne= ctor@0.0.0.0:53656, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 11) Thread[id=3D193, name=3Dqtp643552837-193 Acceptor2 SelectChannelConne= ctor@0.0.0.0:36458, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 12) Thread[id=3D229, name=3Dqtp837657752-229 Acceptor0 SelectChannelConne= ctor@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 13) Thread[id=3D217, name=3Dqtp144414042-217 Acceptor1 SelectChannelConne= ctor@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 14) Thread[id=3D207, name=3Dqtp1236323694-207 Acceptor3 SelectChannelConn= ector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 15) Thread[id=3D218, name=3Dqtp144414042-218 Acceptor2 SelectChannelConne= ctor@0.0.0.0:48493, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 16) Thread[id=3D245, name=3Dqtp149836331-245 Acceptor2 SelectChannelConne= ctor@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 17) Thread[id=3D243, name=3Dqtp149836331-243 Acceptor0 SelectChannelConne= ctor@0.0.0.0:16444, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 18) Thread[id=3D194, name=3Dqtp643552837-194 Acceptor3 SelectChannelConne= ctor@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 19) Thread[id=3D246, name=3Dqtp149836331-246 Acceptor3 SelectChannelConne= ctor@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 20) Thread[id=3D219, name=3Dqtp144414042-219 Acceptor3 SelectChannelConne= ctor@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) =09at __randomizedtesting.SeedInfo.seed([A6F29E4E81D42F0D]:0) FAILED: junit.framework.TestSuite.org.apache.solr.update.SolrCmdDistributo= rTest Error Message: There are still zombie threads that couldn't be terminated: 1) Thread[id= =3D244, name=3Dqtp149836331-244 Acceptor1 SelectChannelConnector@0.0.0.0:16= 444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.n= io.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectCha= nnelConnector.java:104) at org.eclipse.jetty.server.AbstractConnect= or$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org= .eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534= ) at java.lang.Thread.run(Thread.java:679) 2) Thread[id=3D204, n= ame=3Dqtp1236323694-204 Acceptor0 SelectChannelConnector@0.0.0.0:31983, sta= te=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.Se= rverSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) at o= rg.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConn= ector.java:104) at org.eclipse.jetty.server.AbstractConnector$Accep= tor.run(AbstractConnector.java:933) at org.eclipse.jetty.util.threa= d.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse= .jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) = at java.lang.Thread.run(Thread.java:679) 3) Thread[id=3D232, name=3Dqt= p837657752-232 Acceptor3 SelectChannelConnector@0.0.0.0:53656, state=3DBLOC= KED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocke= tChannelImpl.accept(ServerSocketChannelImpl.java:147) at org.eclips= e.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.jav= a:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(A= bstractConnector.java:933) at org.eclipse.jetty.util.thread.QueuedT= hreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java= .lang.Thread.run(Thread.java:679) 4) Thread[id=3D216, name=3Dqtp14441404= 2-216 Acceptor0 SelectChannelConnector@0.0.0.0:48493, state=3DBLOCKED, grou= p=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelI= mpl.accept(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.s= erver.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) = at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractCo= nnector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool= .runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread= .QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thr= ead.run(Thread.java:679) 5) Thread[id=3D205, name=3Dqtp1236323694-205 Ac= ceptor1 SelectChannelConnector@0.0.0.0:31983, state=3DRUNNABLE, group=3DTGR= P-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.acc= ept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(Se= rverSocketChannelImpl.java:163) at org.eclipse.jetty.server.nio.Sel= ectChannelConnector.accept(SelectChannelConnector.java:104) at org.= eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:= 933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThread= Pool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Threa= d.java:679) 6) Thread[id=3D206, name=3Dqtp1236323694-206 Acceptor2 Selec= tChannelConnector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistr= ibutorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSock= etChannelImpl.java:147) at org.eclipse.jetty.server.nio.SelectChann= elConnector.accept(SelectChannelConnector.java:104) at org.eclipse.= jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) = at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadP= ool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.r= un(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:6= 79) 7) Thread[id=3D231, name=3Dqtp837657752-231 Acceptor2 SelectChannelC= onnector@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTes= t] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannel= Impl.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnect= or.accept(SelectChannelConnector.java:104) at org.eclipse.jetty.ser= ver.AbstractConnector$Acceptor.run(AbstractConnector.java:933) at o= rg.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:= 599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(Queued= ThreadPool.java:534) at java.lang.Thread.run(Thread.java:679) 8)= Thread[id=3D191, name=3Dqtp643552837-191 Acceptor0 SelectChannelConnector@= 0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java= :147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept= (SelectChannelConnector.java:104) at org.eclipse.jetty.server.Abstr= actConnector$Acceptor.run(AbstractConnector.java:933) at org.eclips= e.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPoo= l.java:534) at java.lang.Thread.run(Thread.java:679) 9) Thread[i= d=3D192, name=3Dqtp643552837-192 Acceptor1 SelectChannelConnector@0.0.0.0:3= 6458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.= nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectCh= annelConnector.java:104) at org.eclipse.jetty.server.AbstractConnec= tor$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.u= til.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at or= g.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:53= 4) at java.lang.Thread.run(Thread.java:679) 10) Thread[id=3D230, = name=3Dqtp837657752-230 Acceptor1 SelectChannelConnector@0.0.0.0:53656, sta= te=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.S= erverSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerS= ocketChannelImpl.accept(ServerSocketChannelImpl.java:163) at org.ec= lipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector= .java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.r= un(AbstractConnector.java:933) at org.eclipse.jetty.util.thread.Que= uedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jett= y.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at = java.lang.Thread.run(Thread.java:679) 11) Thread[id=3D193, name=3Dqtp6435= 52837-193 Acceptor2 SelectChannelConnector@0.0.0.0:36458, state=3DRUNNABLE,= group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketCha= nnelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelIm= pl.accept(ServerSocketChannelImpl.java:163) at org.eclipse.jetty.se= rver.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) = at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractCon= nector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.= runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.= QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thre= ad.run(Thread.java:679) 12) Thread[id=3D207, name=3Dqtp1236323694-207 Acc= eptor3 SelectChannelConnector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-= SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accep= t(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio= .SelectChannelConnector.accept(SelectChannelConnector.java:104) at = org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.j= ava:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Q= ueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedTh= readPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(T= hread.java:679) 13) Thread[id=3D217, name=3Dqtp144414042-217 Acceptor1 Se= lectChannelConnector@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCmdDi= stributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerS= ocketChannelImpl.java:147) at org.eclipse.jetty.server.nio.SelectCh= annelConnector.accept(SelectChannelConnector.java:104) at org.eclip= se.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) = at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$= 3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.jav= a:679) 14) Thread[id=3D229, name=3Dqtp837657752-229 Acceptor0 SelectChann= elConnector@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributor= Test] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChan= nelImpl.java:147) at org.eclipse.jetty.server.nio.SelectChannelConn= ector.accept(SelectChannelConnector.java:104) at org.eclipse.jetty.= server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) a= t org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.ja= va:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(Que= uedThreadPool.java:534) at java.lang.Thread.run(Thread.java:679) = 15) Thread[id=3D218, name=3Dqtp144414042-218 Acceptor2 SelectChannelConnect= or@0.0.0.0:48493, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] = at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:1= 63) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(S= electChannelConnector.java:104) at org.eclipse.jetty.server.Abstrac= tConnector$Acceptor.run(AbstractConnector.java:933) at org.eclipse.= jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.= java:534) at java.lang.Thread.run(Thread.java:679) 16) Thread[id= =3D245, name=3Dqtp149836331-245 Acceptor2 SelectChannelConnector@0.0.0.0:16= 444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.n= io.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147) = at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectCha= nnelConnector.java:104) at org.eclipse.jetty.server.AbstractConnect= or$Acceptor.run(AbstractConnector.java:933) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org= .eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534= ) at java.lang.Thread.run(Thread.java:679) 17) Thread[id=3D243, n= ame=3Dqtp149836331-243 Acceptor0 SelectChannelConnector@0.0.0.0:16444, stat= e=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.Se= rverSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSo= cketChannelImpl.accept(ServerSocketChannelImpl.java:163) at org.ecl= ipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.= java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.ru= n(AbstractConnector.java:933) at org.eclipse.jetty.util.thread.Queu= edThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty= .util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at j= ava.lang.Thread.run(Thread.java:679) 18) Thread[id=3D194, name=3Dqtp64355= 2837-194 Acceptor3 SelectChannelConnector@0.0.0.0:36458, state=3DBLOCKED, g= roup=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChann= elImpl.accept(ServerSocketChannelImpl.java:147) at org.eclipse.jett= y.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)= at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstrac= tConnector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thr= ead.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.= Thread.run(Thread.java:679) 19) Thread[id=3D246, name=3Dqtp149836331-246 = Acceptor3 SelectChannelConnector@0.0.0.0:16444, state=3DBLOCKED, group=3DTG= RP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.ac= cept(ServerSocketChannelImpl.java:147) at org.eclipse.jetty.server.= nio.SelectChannelConnector.accept(SelectChannelConnector.java:104) = at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnecto= r.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJo= b(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.Queue= dThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.ru= n(Thread.java:679) 20) Thread[id=3D219, name=3Dqtp144414042-219 Acceptor3= SelectChannelConnector@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCm= dDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(Serv= erSocketChannelImpl.java:147) at org.eclipse.jetty.server.nio.Selec= tChannelConnector.accept(SelectChannelConnector.java:104) at org.ec= lipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:93= 3) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedT= hreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPo= ol$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.= java:679) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie = threads that couldn't be terminated: 1) Thread[id=3D244, name=3Dqtp149836331-244 Acceptor1 SelectChannelConne= ctor@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 2) Thread[id=3D204, name=3Dqtp1236323694-204 Acceptor0 SelectChannelConn= ector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 3) Thread[id=3D232, name=3Dqtp837657752-232 Acceptor3 SelectChannelConne= ctor@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 4) Thread[id=3D216, name=3Dqtp144414042-216 Acceptor0 SelectChannelConne= ctor@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 5) Thread[id=3D205, name=3Dqtp1236323694-205 Acceptor1 SelectChannelConn= ector@0.0.0.0:31983, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 6) Thread[id=3D206, name=3Dqtp1236323694-206 Acceptor2 SelectChannelConn= ector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 7) Thread[id=3D231, name=3Dqtp837657752-231 Acceptor2 SelectChannelConne= ctor@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 8) Thread[id=3D191, name=3Dqtp643552837-191 Acceptor0 SelectChannelConne= ctor@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 9) Thread[id=3D192, name=3Dqtp643552837-192 Acceptor1 SelectChannelConne= ctor@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 10) Thread[id=3D230, name=3Dqtp837657752-230 Acceptor1 SelectChannelConne= ctor@0.0.0.0:53656, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 11) Thread[id=3D193, name=3Dqtp643552837-193 Acceptor2 SelectChannelConne= ctor@0.0.0.0:36458, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 12) Thread[id=3D207, name=3Dqtp1236323694-207 Acceptor3 SelectChannelConn= ector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 13) Thread[id=3D217, name=3Dqtp144414042-217 Acceptor1 SelectChannelConne= ctor@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 14) Thread[id=3D229, name=3Dqtp837657752-229 Acceptor0 SelectChannelConne= ctor@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 15) Thread[id=3D218, name=3Dqtp144414042-218 Acceptor2 SelectChannelConne= ctor@0.0.0.0:48493, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 16) Thread[id=3D245, name=3Dqtp149836331-245 Acceptor2 SelectChannelConne= ctor@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 17) Thread[id=3D243, name=3Dqtp149836331-243 Acceptor0 SelectChannelConne= ctor@0.0.0.0:16444, state=3DRUNNABLE, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:163) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 18) Thread[id=3D194, name=3Dqtp643552837-194 Acceptor3 SelectChannelConne= ctor@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 19) Thread[id=3D246, name=3Dqtp149836331-246 Acceptor3 SelectChannelConne= ctor@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) 20) Thread[id=3D219, name=3Dqtp144414042-219 Acceptor3 SelectChannelConne= ctor@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-SolrCmdDistributorTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:147) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(Selec= tChannelConnector.java:104) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:933) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:534) at java.lang.Thread.run(Thread.java:679) =09at __randomizedtesting.SeedInfo.seed([A6F29E4E81D42F0D]:0) Build Log: [...truncated 7858 lines...] [junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest [junit4:junit4] 2> 4 T103 oas.SolrTestCaseJ4.setUp ###Starting testDistri= bSearch [junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/worksp= ace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./solrtest-= RecoveryZkTest-1351323457217 [junit4:junit4] 2> 11 T103 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4:junit4] 2> 94 T104 oaz.Environment.logEnv Server environment:zook= eeper.version=3D3.3.6-1366786, built on 07/29/2012 06:22 GMT [junit4:junit4] 2> 94 T104 oaz.Environment.logEnv Server environment:host= .name=3Dlucene.zones.apache.org [junit4:junit4] 2> 94 T104 oaz.Environment.logEnv Server environment:java= .version=3D1.6.0_32 [junit4:junit4] 2> 94 T104 oaz.Environment.logEnv Server environment:java= .vendor=3DSun Microsystems Inc. [junit4:junit4] 2> 95 T104 oaz.Environment.logEnv Server environment:java= .home=3D/usr/local/openjdk6/jre [junit4:junit4] 2> 95 T104 oaz.Environment.logEnv Server environment:java= .class.path=3D/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/w= orkspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-test-framework/clas= ses/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-t= runk/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspac= e/Lucene-Solr-NightlyTests-trunk/lucene/build/test-framework/classes/java:/= usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucen= e/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-= Solr-NightlyTests-trunk/solr/build/solr-solrj/classes/java:/usr/home/hudson= /hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core= /classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/lucene/build/analysis/common/lucene-analyzers-common-5.0-SNAPSHOT= .jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk= /lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-5.0-SNAPSHOT.jar:= /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/luce= ne/build/analysis/phonetic/lucene-analyzers-phonetic-5.0-SNAPSHOT.jar:/usr/= home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/bu= ild/highlighter/lucene-highlighter-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson= -slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/memory/lucene-= memory-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr= -NightlyTests-trunk/lucene/build/misc/lucene-misc-5.0-SNAPSHOT.jar:/usr/hom= e/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build= /spatial/lucene-spatial-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/work= space/Lucene-Solr-NightlyTests-trunk/lucene/build/suggest/lucene-suggest-5.= 0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyT= ests-trunk/lucene/build/grouping/lucene-grouping-5.0-SNAPSHOT.jar:/usr/home= /hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/= queries/lucene-queries-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/works= pace/Lucene-Solr-NightlyTests-trunk/lucene/build/queryparser/lucene-querypa= rser-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-N= ightlyTests-trunk/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson= -slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-codec= -1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-t= runk/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-sla= ve/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-lang-2.6.= jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/= solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Luce= ne-Solr-NightlyTests-trunk/solr/core/lib/guava-r05.jar:/usr/home/hudson/hud= son-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/javax.serv= let-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Night= lyTests-trunk/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave= /workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/commons-io-2.1.jar= :/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/sol= r/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hu= dson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/h= ttpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Night= lyTests-trunk/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/huds= on-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/log4j-over= -slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nightl= yTests-trunk/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-sla= ve/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/slf4j-jdk14-1.6.= 4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trun= k/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace= /Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/hom= e/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/example= /lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/w= orkspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/jetty-deploy-8.1.2= .v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyT= ests-trunk/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson= /hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/jet= ty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-So= lr-NightlyTests-trunk/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/h= ome/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/examp= le/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/wor= kspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/jetty-server-8.1.2.v= 20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTes= ts-trunk/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudso= n/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/je= tty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene= -Solr-NightlyTests-trunk/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:= /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr= /example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/wo= rkspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/servlet-api-3.0.jar= :/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/sol= r/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/huds= on-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/classes= /java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trun= k/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/wo= rkspace/Lucene-Solr-NightlyTests-trunk/lucene/test-framework/lib/randomized= testing-runner-2.0.3.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/= ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-sl= ave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave= /tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/= ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.= 8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/l= ib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apac= he-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-= logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xa= lan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.j= ar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/h= udson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hud= son-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/t= ools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant= /ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/l= ib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/h= ome/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/h= udson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-= slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/to= ols/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/too= ls/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/hom= e/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/jun= it4-ant-2.0.3.jar [junit4:junit4] 2> 96 T104 oaz.Environment.logEnv Server environment:java= .library.path=3D/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk= 6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib= /amd64:/lib:/usr/lib:/usr/local/lib [junit4:junit4] 2> 96 T104 oaz.Environment.logEnv Server environment:java= .io.tmpdir=3D. [junit4:junit4] 2> 96 T104 oaz.Environment.logEnv Server environment:java= .compiler=3D [junit4:junit4] 2> 96 T104 oaz.Environment.logEnv Server environment:os.n= ame=3DFreeBSD [junit4:junit4] 2> 96 T104 oaz.Environment.logEnv Server environment:os.a= rch=3Damd64 [junit4:junit4] 2> 96 T104 oaz.Environment.logEnv Server environment:os.v= ersion=3D9.0-RELEASE [junit4:junit4] 2> 97 T104 oaz.Environment.logEnv Server environment:user= .name=3Dhudson [junit4:junit4] 2> 97 T104 oaz.Environment.logEnv Server environment:user= .home=3D/home/hudson [junit4:junit4] 2> 97 T104 oaz.Environment.logEnv Server environment:user= .dir=3D/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-tru= nk/solr/build/solr-core/test/J1 [junit4:junit4] 2> 131 T104 oazs.ZooKeeperServer.setTickTime tickTime set= to 1000 [junit4:junit4] 2> 159 T104 oazs.NIOServerCnxn$Factory. binding to = port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 212 T104 oazsp.FileTxnSnapLog.save Snapshotting: 0 [junit4:junit4] 2> 218 T103 oasc.ZkTestServer.run start zk server on port= :51336 [junit4:junit4] 2> 323 T103 oaz.Environment.logEnv Client environment:zoo= keeper.version=3D3.3.6-1366786, built on 07/29/2012 06:22 GMT [junit4:junit4] 2> 324 T103 oaz.Environment.logEnv Client environment:hos= t.name=3Dlucene.zones.apache.org [junit4:junit4] 2> 324 T103 oaz.Environment.logEnv Client environment:jav= a.version=3D1.6.0_32 [junit4:junit4] 2> 324 T103 oaz.Environment.logEnv Client environment:jav= a.vendor=3DSun Microsystems Inc. [junit4:junit4] 2> 324 T103 oaz.Environment.logEnv Client environment:jav= a.home=3D/usr/local/openjdk6/jre [junit4:junit4] 2> 325 T103 oaz.Environment.logEnv Client environment:jav= a.class.path=3D/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyT= ests-trunk/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/= workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-test-framework/cla= sses/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-= trunk/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspa= ce/Lucene-Solr-NightlyTests-trunk/lucene/build/test-framework/classes/java:= /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/luce= ne/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene= -Solr-NightlyTests-trunk/solr/build/solr-solrj/classes/java:/usr/home/hudso= n/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-cor= e/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyT= ests-trunk/lucene/build/analysis/common/lucene-analyzers-common-5.0-SNAPSHO= T.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trun= k/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-5.0-SNAPSHOT.jar= :/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/luc= ene/build/analysis/phonetic/lucene-analyzers-phonetic-5.0-SNAPSHOT.jar:/usr= /home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/b= uild/highlighter/lucene-highlighter-5.0-SNAPSHOT.jar:/usr/home/hudson/hudso= n-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/memory/lucene= -memory-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Sol= r-NightlyTests-trunk/lucene/build/misc/lucene-misc-5.0-SNAPSHOT.jar:/usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/buil= d/spatial/lucene-spatial-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/wor= kspace/Lucene-Solr-NightlyTests-trunk/lucene/build/suggest/lucene-suggest-5= .0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nightly= Tests-trunk/lucene/build/grouping/lucene-grouping-5.0-SNAPSHOT.jar:/usr/hom= e/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build= /queries/lucene-queries-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/work= space/Lucene-Solr-NightlyTests-trunk/lucene/build/queryparser/lucene-queryp= arser-5.0-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-= NightlyTests-trunk/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudso= n-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-code= c-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-= trunk/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-sl= ave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/commons-lang-2.6= .jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk= /solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Luc= ene-Solr-NightlyTests-trunk/solr/core/lib/guava-r05.jar:/usr/home/hudson/hu= dson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/javax.ser= vlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nigh= tlyTests-trunk/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slav= e/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/commons-io-2.1.ja= r:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/so= lr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/L= ucene-Solr-NightlyTests-trunk/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/h= udson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/= httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nigh= tlyTests-trunk/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hud= son-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/log4j-ove= r-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Night= lyTests-trunk/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-sl= ave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/slf4j-jdk14-1.6= .4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-tru= nk/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspac= e/Lucene-Solr-NightlyTests-trunk/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/exampl= e/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/= workspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/jetty-deploy-8.1.= 2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nightly= Tests-trunk/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudso= n/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/je= tty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-S= olr-NightlyTests-trunk/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/= home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/exam= ple/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/wo= rkspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/jetty-server-8.1.2.= v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/huds= on/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/j= etty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucen= e-Solr-NightlyTests-trunk/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar= :/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/sol= r/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/w= orkspace/Lucene-Solr-NightlyTests-trunk/solr/example/lib/servlet-api-3.0.ja= r:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/so= lr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hud= son-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/classe= s/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-tru= nk/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/w= orkspace/Lucene-Solr-NightlyTests-trunk/lucene/test-framework/lib/randomize= dtesting-runner-2.0.3.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib= /ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-s= lave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slav= e/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant= /ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1= .8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/= lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apa= che-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons= -logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-x= alan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.= jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/= hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hu= dson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/= tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/An= t/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/= lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/= home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/= hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson= -slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/t= ools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/to= ols/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/ho= me/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/ju= nit4-ant-2.0.3.jar [junit4:junit4] 2> 326 T103 oaz.Environment.logEnv Client environment:jav= a.library.path=3D/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjd= k6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/li= b/amd64:/lib:/usr/lib:/usr/local/lib [junit4:junit4] 2> 326 T103 oaz.Environment.logEnv Client environment:jav= a.io.tmpdir=3D. [junit4:junit4] 2> 326 T103 oaz.Environment.logEnv Client environment:jav= a.compiler=3D [junit4:junit4] 2> 327 T103 oaz.Environment.logEnv Client environment:os.= name=3DFreeBSD [junit4:junit4] 2> 327 T103 oaz.Environment.logEnv Client environment:os.= arch=3Damd64 [junit4:junit4] 2> 327 T103 oaz.Environment.logEnv Client environment:os.= version=3D9.0-RELEASE [junit4:junit4] 2> 327 T103 oaz.Environment.logEnv Client environment:use= r.name=3Dhudson [junit4:junit4] 2> 328 T103 oaz.Environment.logEnv Client environment:use= r.home=3D/home/hudson [junit4:junit4] 2> 328 T103 oaz.Environment.logEnv Client environment:use= r.dir=3D/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-tr= unk/solr/build/solr-core/test/J1 [junit4:junit4] 2> 333 T103 oaz.ZooKeeper. Initiating client connec= tion, connectString=3D127.0.0.1:51336 sessionTimeout=3D10000 watcher=3Dorg.= apache.solr.common.cloud.ConnectionManager@76bb9f5b [junit4:junit4] 2> 392 T109 oaz.ClientCnxn$SendThread.startConnect Openin= g socket connection to server /127.0.0.1:51336 [junit4:junit4] 2> 392 T103 oascc.ConnectionManager.waitForConnected Wait= ing for client to connect to ZooKeeper [junit4:junit4] 2> 396 T109 oaz.ClientCnxn$SendThread.primeConnection Soc= ket connection established to localhost/127.0.0.1:51336, initiating session [junit4:junit4] 2> 397 T105 oazs.NIOServerCnxn$Factory.run Accepted socke= t connection from /140.211.11.196:57930 [junit4:junit4] 2> 411 T105 oazs.NIOServerCnxn.readConnectRequest Client = attempting to establish new session at /140.211.11.196:57930 [junit4:junit4] 2> 425 T107 oazsp.FileTxnLog.append Creating new log file= : log.1 [junit4:junit4] 2> 500 T107 oazs.NIOServerCnxn.finishSessionInit Establis= hed session 0x13aa12797c40000 with negotiated timeout 10000 for client /140= .211.11.196:57930 [junit4:junit4] 2> 501 T109 oaz.ClientCnxn$SendThread.readConnectResult S= ession establishment complete on server localhost/127.0.0.1:51336, sessioni= d =3D 0x13aa12797c40000, negotiated timeout =3D 10000 [junit4:junit4] 2> 506 T110 oascc.ConnectionManager.process Watcher org.a= pache.solr.common.cloud.ConnectionManager@76bb9f5b name:ZooKeeperConnection= Watcher:127.0.0.1:51336 got event WatchedEvent state:SyncConnected type:No= ne path:null path:null type:None [junit4:junit4] 2> 506 T103 oascc.ConnectionManager.waitForConnected Clie= nt is connected to ZooKeeper [junit4:junit4] 2> 510 T103 oascc.SolrZkClient.makePath makePath: /solr [junit4:junit4] 2> 595 T108 oazs.PrepRequestProcessor.pRequest Processed = session termination for sessionid: 0x13aa12797c40000 [junit4:junit4] 2> 596 T110 oaz.ClientCnxn$EventThread.run EventThread sh= ut down [junit4:junit4] 2> 596 T103 oaz.ZooKeeper.close Session: 0x13aa12797c4000= 0 closed [junit4:junit4] 2> 597 T103 oaz.ZooKeeper. Initiating client connec= tion, connectString=3D127.0.0.1:51336/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@4275b35 [junit4:junit4] 2> 597 T105 oazs.NIOServerCnxn.closeSock Closed socket co= nnection for client /140.211.11.196:57930 which had sessionid 0x13aa12797c4= 0000 [junit4:junit4] 2> 598 T111 oaz.ClientCnxn$SendThread.startConnect Openin= g socket connection to server /127.0.0.1:51336 [junit4:junit4] 2> 598 T103 oascc.ConnectionManager.waitForConnected Wait= ing for client to connect to ZooKeeper [junit4:junit4] 2> 599 T111 oaz.ClientCnxn$SendThread.primeConnection Soc= ket connection established to localhost/127.0.0.1:51336, initiating session [junit4:junit4] 2> 599 T105 oazs.NIOServerCnxn$Factory.run Accepted socke= t connection from /140.211.11.196:34520 [junit4:junit4] 2> 600 T105 oazs.NIOServerCnxn.readConnectRequest Client = attempting to establish new session at /140.211.11.196:34520 [junit4:junit4] 2> 601 T107 oazs.NIOServerCnxn.finishSessionInit Establis= hed session 0x13aa12797c40001 with negotiated timeout 10000 for client /140= .211.11.196:34520 [junit4:junit4] 2> 601 T111 oaz.ClientCnxn$SendThread.readConnectResult S= ession establishment complete on server localhost/127.0.0.1:51336, sessioni= d =3D 0x13aa12797c40001, negotiated timeout =3D 10000 [junit4:junit4] 2> 601 T112 oascc.ConnectionManager.process Watcher org.a= pache.solr.common.cloud.ConnectionManager@4275b35 name:ZooKeeperConnection = Watcher:127.0.0.1:51336/solr got event WatchedEvent state:SyncConnected typ= e:None path:null path:null type:None [junit4:junit4] 2> 602 T103 oascc.ConnectionManager.waitForConnected Clie= nt is connected to ZooKeeper [junit4:junit4] 2> 606 T103 oascc.SolrZkClient.makePath makePath: /collec= tions/collection1 [junit4:junit4] 2> 610 T103 oascc.SolrZkClient.makePath makePath: /collec= tions/collection1/shards [junit4:junit4] 2> 619 T103 oascc.SolrZkClient.makePath makePath: /collec= tions/control_collection [junit4:junit4] 2> 622 T103 oascc.SolrZkClient.makePath makePath: /collec= tions/control_collection/shards [junit4:junit4] 2> 625 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1= /solrconfig.xml [junit4:junit4] 2> 644 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/solrconfig.xml [junit4:junit4] 2> 659 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/sch= ema.xml [junit4:junit4] 2> 660 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/schema.xml [junit4:junit4] 2> 764 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1= /solrconfig.xml [junit4:junit4] 2> 765 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/solrconfig.xml [junit4:junit4] 2> 778 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/= stopwords.txt [junit4:junit4] 2> 779 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/stopwords.txt [junit4:junit4] 2> 782 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/= protwords.txt [junit4:junit4] 2> 782 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/protwords.txt [junit4:junit4] 2> 786 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/c= urrency.xml [junit4:junit4] 2> 786 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/currency.xml [junit4:junit4] 2> 801 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /con= figs/conf1/open-exchange-rates.json [junit4:junit4] 2> 802 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/open-exchange-rates.json [junit4:junit4] 2> 805 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /= configs/conf1/mapping-ISOLatin1Accent.txt [junit4:junit4] 2> 806 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/mapping-ISOLatin1Accent.txt [junit4:junit4] 2> 810 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/con= f1/old_synonyms.txt [junit4:junit4] 2> 811 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/old_synonyms.txt [junit4:junit4] 2> 814 T103 oasc.AbstractZkTestCase.putConfig put /usr/ho= me/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/= solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/s= ynonyms.txt [junit4:junit4] 2> 815 T103 oascc.SolrZkClient.makePath makePath: /config= s/conf1/synonyms.txt [junit4:junit4] 2> 819 T108 oazs.PrepRequestProcessor.pRequest Processed = session termination for sessionid: 0x13aa12797c40001 [junit4:junit4] 2> 821 T112 oaz.ClientCnxn$EventThread.run EventThread sh= ut down [junit4:junit4] 2> 821 T105 oazs.NIOServerCnxn.closeSock Closed socket co= nnection for client /140.211.11.196:34520 which had sessionid 0x13aa12797c4= 0001 [junit4:junit4] 2> 821 T103 oaz.ZooKeeper.close Session: 0x13aa12797c4000= 1 closed [junit4:junit4] 2> 1343 T103 oejs.Server.doStart jetty-8.1.2.v20120308 [junit4:junit4] 2> 1596 T103 oejs.AbstractConnector.doStart Started Selec= tChannelConnector@0.0.0.0:60207 [junit4:junit4] 2> 1625 T103 oasc.SolrResourceLoader.locateSolrHome JNDI = not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 1626 T103 oasc.SolrResourceLoader.locateSolrHome using= system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-controljetty-1351323458034 [junit4:junit4] 2> 1627 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/L= ucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.sol= r.cloud.RecoveryZkTest-controljetty-1351323458034/' [junit4:junit4] 2> 1658 T103 oass.SolrDispatchFilter.init SolrDispatchFil= ter.init() [junit4:junit4] 2> 1659 T103 oasc.SolrResourceLoader.locateSolrHome JNDI = not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 1660 T103 oasc.SolrResourceLoader.locateSolrHome using= system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-controljetty-1351323458034 [junit4:junit4] 2> 1662 T103 oasc.CoreContainer$Initializer.initialize lo= oking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nig= htlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Recove= ryZkTest-controljetty-1351323458034/solr.xml [junit4:junit4] 2> 1663 T103 oasc.CoreContainer. New CoreContainer = 1431908901 [junit4:junit4] 2> 1664 T103 oasc.CoreContainer.load Loading CoreContaine= r using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nig= htlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Recove= ryZkTest-controljetty-1351323458034/' [junit4:junit4] 2> 1666 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-So= lr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.= RecoveryZkTest-controljetty-1351323458034/' [junit4:junit4] 2> 1697 T103 oasc.CoreContainer.load Registering Log List= ener [junit4:junit4] 2> 1709 T103 oasc.CoreContainer.initZooKeeper Zookeeper c= lient=3D127.0.0.1:51336/solr [junit4:junit4] 2> 1734 T103 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:51336/solr sessionTimeout=3D5000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@266286e3 [junit4:junit4] 2> 1735 T123 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:51336 [junit4:junit4] 2> 1735 T103 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 1737 T123 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost/127.0.0.1:51336, initiating sessio= n [junit4:junit4] 2> 1737 T105 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /140.211.11.196:50379 [junit4:junit4] 2> 1738 T105 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /140.211.11.196:50379 [junit4:junit4] 2> 1765 T107 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13aa12797c40002 with negotiated timeout 5000 for client /140= .211.11.196:50379 [junit4:junit4] 2> 1765 T123 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost/127.0.0.1:51336, session= id =3D 0x13aa12797c40002, negotiated timeout =3D 5000 [junit4:junit4] 2> 1766 T124 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@266286e3 name:ZooKeeperConnectio= n Watcher:127.0.0.1:51336/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 1766 T103 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 1781 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1791 T103 oascc.SolrZkClient.makePath makePath: /live_= nodes [junit4:junit4] 2> 1794 T103 oasc.ZkController.createEphemeralLiveNode Re= gister node as live in ZooKeeper:/live_nodes/127.0.0.1:60207_solr [junit4:junit4] 2> 1798 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:dele= te cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/live_nodes/127.0.0.1:60207_solr Error:KeeperErrorCode =3D NoNode for = /solr/live_nodes/127.0.0.1:60207_solr [junit4:junit4] 2> 1800 T103 oascc.SolrZkClient.makePath makePath: /live_= nodes/127.0.0.1:60207_solr [junit4:junit4] 2> 1804 T103 oashc.HttpShardHandlerFactory.getParameter S= etting socketTimeout to: 0 [junit4:junit4] 2> 1805 T103 oashc.HttpShardHandlerFactory.getParameter S= etting urlScheme to: http:// [junit4:junit4] 2> 1805 T103 oashc.HttpShardHandlerFactory.getParameter S= etting connTimeout to: 0 [junit4:junit4] 2> 1806 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 1807 T103 oashc.HttpShardHandlerFactory.getParameter S= etting corePoolSize to: 0 [junit4:junit4] 2> 1807 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 1808 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maxThreadIdleTime to: 5 [junit4:junit4] 2> 1809 T103 oashc.HttpShardHandlerFactory.getParameter S= etting sizeOfQueue to: -1 [junit4:junit4] 2> 1809 T103 oashc.HttpShardHandlerFactory.getParameter S= etting fairnessPolicy to: false [junit4:junit4] 2> 1810 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000&s= ocketTimeout=3D0&connTimeout=3D0&retry=3Dfalse [junit4:junit4] 2> 1826 T103 oascc.SolrZkClient.makePath makePath: /overs= eer_elect/election [junit4:junit4] 2> 1850 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:dele= te cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ove= rseer_elect/leader [junit4:junit4] 2> 1851 T103 oascc.SolrZkClient.makePath makePath: /overs= eer_elect/leader [junit4:junit4] 2> 1860 T103 oasc.Overseer.start Overseer (id=3D885603341= 09147138-127.0.0.1:60207_solr-n_0000000000) starting [junit4:junit4] 2> 1869 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1879 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1881 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1885 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1887 T126 oasc.OverseerCollectionProcessor.run Process= current queue of collection creations [junit4:junit4] 2> 1887 T103 oascc.SolrZkClient.makePath makePath: /clust= erstate.json [junit4:junit4] 2> 1900 T103 oascc.ZkStateReader.createClusterStateWatche= rsAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 1920 T125 oasc.Overseer$ClusterStateUpdater.run Starti= ng to work on the main queue [junit4:junit4] 2> 1951 T103 oasc.CoreContainer.create Creating SolrCore = 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-controljetty-1351323458034/collection1 [junit4:junit4] 2> 1952 T103 oasc.ZkController.createCollectionZkNode Che= ck for collection zkNode:control_collection [junit4:junit4] 2> 1953 T103 oasc.ZkController.createCollectionZkNode Col= lection zkNode exists [junit4:junit4] 2> 1953 T103 oasc.ZkController.readConfigName Load collec= tion config from:/collections/control_collection [junit4:junit4] 2> 1955 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-So= lr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.= RecoveryZkTest-controljetty-1351323458034/collection1/' [junit4:junit4] 2> 1956 T103 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTest= s-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-c= ontroljetty-1351323458034/collection1/lib/classes/' to classloader [junit4:junit4] 2> 1957 T103 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTest= s-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-c= ontroljetty-1351323458034/collection1/lib/README' to classloader [junit4:junit4] 2> 2026 T103 oasc.SolrConfig. Using Lucene MatchVer= sion: LUCENE_50 [junit4:junit4] 2> 2084 T103 oasc.SolrConfig. Loaded SolrConfig: so= lrconfig.xml [junit4:junit4] 2> 2186 T103 oass.IndexSchema.readSchema Reading Solr Sch= ema [junit4:junit4] 2> 2196 T103 oass.IndexSchema.readSchema Schema name=3Dte= st [junit4:junit4] 2> 2608 T103 oass.OpenExchangeRatesOrgProvider.init Initi= alized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 2617 T103 oass.IndexSchema.readSchema default search f= ield in schema is text [junit4:junit4] 2> 2620 T103 oass.IndexSchema.readSchema unique key field= : id [junit4:junit4] 2> 2637 T103 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 2642 T103 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 2646 T103 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 2649 T103 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 2652 T103 oasc.SolrCore. [collection1] Opening n= ew SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyT= ests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkT= est-controljetty-1351323458034/collection1/, dataDir=3D./org.apache.solr.cl= oud.RecoveryZkTest-1351323457216/control/data/ [junit4:junit4] 2> 2652 T103 oasc.JmxMonitoredMap. JMX monitoring i= s enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBe= anServer@56278e83 [junit4:junit4] 2> 2653 T103 oasc.SolrCore.initDirectoryFactory org.apach= e.solr.core.MockDirectoryFactory [junit4:junit4] 2> 2654 T103 oasc.CachingDirectoryFactory.injectLockFacto= ry WARNING No lockType configured for MockDirWrapper(org.apache.lucene.stor= e.RAMDirectory@381f7fbe lockFactory=3Dorg.apache.lucene.store.SingleInstanc= eLockFactory@1bfbafbb) assuming 'simple' [junit4:junit4] 2> 2654 T103 oasc.CachingDirectoryFactory.get return new = directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTe= st-1351323457216/control/data forceNew:false [junit4:junit4] 2> 2655 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/control/data [junit4:junit4] 2> 2655 T103 oasc.SolrCore.getNewIndexDir New index direc= tory detected: old=3Dnull new=3D./org.apache.solr.cloud.RecoveryZkTest-1351= 323457216/control/data/index/ [junit4:junit4] 2> 2656 T103 oasc.SolrCore.initIndex WARNING [collection1= ] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-135132345721= 6/control/data/index' doesn't exist. Creating new index... [junit4:junit4] 2> 2657 T103 oasc.CachingDirectoryFactory.get return new = directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTe= st-1351323457216/control/data/index forceNew:false [junit4:junit4] 2> 2660 T103 oasc.SolrDeletionPolicy.onCommit SolrDeletio= nPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3DMockDirWrapper(org.apache.lucene.st= ore.RAMDirectory@d9438de lockFactory=3Dorg.apache.lucene.store.NativeFSLock= Factory@48c91629),segFN=3Dsegments_1,generation=3D1,filenames=3D[segments_1= ] [junit4:junit4] 2> 2660 T103 oasc.SolrDeletionPolicy.updateCommits newest= commit =3D 1[segments_1] [junit4:junit4] 2> 2661 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/control/data/index [junit4:junit4] 2> 2661 T103 oasc.SolrCore.initWriters created xml: solr.= XMLResponseWriter [junit4:junit4] 2> 2663 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "de= dupe" [junit4:junit4] 2> 2664 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "de= dupe-allfields" [junit4:junit4] 2> 2664 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "st= ored_sig" [junit4:junit4] 2> 2665 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "un= iq-fields" [junit4:junit4] 2> 2665 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "di= strib-dup-test-chain-implicit" [junit4:junit4] 2> 2666 T103 oasc.RequestHandlers.initHandlersFromConfig = adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 2666 T103 oasc.RequestHandlers.initHandlersFromConfig = created /replication: solr.ReplicationHandler [junit4:junit4] 2> 2667 T103 oasc.RequestHandlers.initHandlersFromConfig = created standard: solr.StandardRequestHandler [junit4:junit4] 2> 2668 T103 oasc.RequestHandlers.initHandlersFromConfig = created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 2669 T103 oasc.RequestHandlers.initHandlersFromConfig = created dismax: solr.SearchHandler [junit4:junit4] 2> 2669 T103 oasc.RequestHandlers.initHandlersFromConfig = created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler [junit4:junit4] 2> 2670 T103 oasc.RequestHandlers.initHandlersFromConfig = created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 2671 T103 oasc.RequestHandlers.initHandlersFromConfig = created defaults: solr.StandardRequestHandler [junit4:junit4] 2> 2671 T103 oasc.RequestHandlers.initHandlersFromConfig = adding lazy requestHandler: solr.StandardRequestHandler [junit4:junit4] 2> 2672 T103 oasc.RequestHandlers.initHandlersFromConfig = created lazy: solr.StandardRequestHandler [junit4:junit4] 2> 2672 T103 oasc.RequestHandlers.initHandlersFromConfig = created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 2673 T103 oasc.RequestHandlers.initHandlersFromConfig = created /terms: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 2674 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 2674 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHa= ndler [junit4:junit4] 2> 2675 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHa= ndler [junit4:junit4] 2> 2676 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.S= earchHandler [junit4:junit4] 2> 2677 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 2677 T103 oasc.RequestHandlers.initHandlersFromConfig = created tvrh: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 2678 T103 oasc.RequestHandlers.initHandlersFromConfig = created /mlt: solr.MoreLikeThisHandler [junit4:junit4] 2> 2679 T103 oasc.RequestHandlers.initHandlersFromConfig = created /debug/dump: solr.DumpRequestHandler [junit4:junit4] 2> 2684 T103 oashl.XMLLoader.init xsltCacheLifetimeSecond= s=3D60 [junit4:junit4] 2> 2688 T103 oasc.SolrCore.initDeprecatedSupport WARNING = solrconfig.xml uses deprecated , Please update your co= nfig to use the ShowFileRequestHandler. [junit4:junit4] 2> 2690 T103 oasc.SolrCore.initDeprecatedSupport WARNING = adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.= TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, = CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT] [junit4:junit4] 2> 2692 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/control/data [junit4:junit4] 2> 2693 T103 oass.SolrIndexSearcher. Opening Search= er@7a49182 main [junit4:junit4] 2> 2694 T103 oasu.CommitTracker. Hard AutoCommit: d= isabled [junit4:junit4] 2> 2694 T103 oasu.CommitTracker. Soft AutoCommit: d= isabled [junit4:junit4] 2> 2695 T103 oashc.SpellCheckComponent.inform Initializin= g spell checkers [junit4:junit4] 2> 2730 T103 oass.DirectSolrSpellChecker.init init: {name= =3Ddirect,classname=3DDirectSolrSpellChecker,field=3Dlowerfilt,minQueryLeng= th=3D3} [junit4:junit4] 2> 2786 T131 oasc.SolrCore.registerSearcher [collection1]= Registered new searcher Searcher@7a49182 main{StandardDirectoryReader(segm= ents_1:1)} [junit4:junit4] 2> 2798 T103 oasc.ZkController.publish numShards not foun= d on descriptor - reading it from system property [junit4:junit4] 2> 2799 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x53 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer/q= ueue [junit4:junit4] 2> 2812 T103 oasc.CoreContainer.register registering core= : collection1 [junit4:junit4] 2> 2812 T103 oasc.ZkController.register Register shard - = core:collection1 address:http://127.0.0.1:60207/solr shardId:control_shard [junit4:junit4] 2> 2829 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnections=3D10000&maxConnectionsPerHost=3D20&c= onnTimeout=3D30000&socketTimeout=3D30000&retry=3Dfalse [junit4:junit4] 2> 2846 T103 oascc.SolrZkClient.makePath makePath: /colle= ctions/control_collection/leader_elect/control_shard/election [junit4:junit4] 2> 2874 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:dele= te cxid:0x62 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/collections/control_collection/leaders Error:KeeperErrorCode =3D NoN= ode for /solr/collections/control_collection/leaders [junit4:junit4] 2> 2875 T103 oasc.ShardLeaderElectionContext.runLeaderPro= cess Running the leader process. [junit4:junit4] 2> 2876 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x63 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 2928 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 2929 T125 oasc.Overseer$ClusterStateUpdater.updateStat= e Update state numShards=3Dnull message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":null, [junit4:junit4] 2> =09 "shard":"control_shard", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"control_collection", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:60207_solr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:60207/solr"} [junit4:junit4] 2> 2940 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x6a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/overs= eer/queue-work [junit4:junit4] 2> 2958 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (1) [junit4:junit4] 2> 3380 T103 oasc.ShardLeaderElectionContext.waitForRepli= casToComeUp Enough replicas found to continue. [junit4:junit4] 2> 3380 T103 oasc.ShardLeaderElectionContext.runLeaderPro= cess I may be the new leader - try and sync [junit4:junit4] 2> 3381 T103 oasc.SyncStrategy.sync Sync replicas to http= ://127.0.0.1:60207/solr/collection1/ [junit4:junit4] 2> 3381 T103 oasc.SyncStrategy.syncReplicas Sync Success = - now sync replicas to me [junit4:junit4] 2> 3381 T103 oasc.SyncStrategy.syncToMe http://127.0.0.1:= 60207/solr/collection1/ has no replicas [junit4:junit4] 2> 3382 T103 oasc.ShardLeaderElectionContext.runLeaderPro= cess I am the new leader: http://127.0.0.1:60207/solr/collection1/ [junit4:junit4] 2> 3382 T103 oascc.SolrZkClient.makePath makePath: /colle= ctions/control_collection/leaders/control_shard [junit4:junit4] 2> 3389 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40002 type:crea= te cxid:0x89 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 3465 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 3478 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (1) [junit4:junit4] 2> 3503 T103 oasc.ZkController.register We are http://127= .0.0.1:60207/solr/collection1/ and leader is http://127.0.0.1:60207/solr/co= llection1/ [junit4:junit4] 2> 3503 T103 oasc.ZkController.register No LogReplay need= ed for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:60207/solr [junit4:junit4] 2> 3503 T103 oasc.ZkController.checkRecovery I am the lea= der, no recovery necessary [junit4:junit4] 2> 3504 T103 oasc.ZkController.publish numShards not foun= d on descriptor - reading it from system property [junit4:junit4] 2> 3507 T103 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 3508 T103 oass.SolrDispatchFilter.init user.dir=3D/usr= /home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/bui= ld/solr-core/test/J1 [junit4:junit4] 2> 3509 T103 oass.SolrDispatchFilter.init SolrDispatchFil= ter.init() done [junit4:junit4] 2> 3563 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&fol= lowRedirects=3Dfalse [junit4:junit4] 2> 3578 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config: [junit4:junit4] 2> 3588 T103 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:51336/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@63c9982e [junit4:junit4] 2> 3589 T103 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 3589 T132 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:51336 [junit4:junit4] 2> 3590 T132 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost/127.0.0.1:51336, initiating sessio= n [junit4:junit4] 2> 3590 T105 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /140.211.11.196:26054 [junit4:junit4] 2> 3591 T105 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /140.211.11.196:26054 [junit4:junit4] 2> 3593 T107 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13aa12797c40003 with negotiated timeout 10000 for client /14= 0.211.11.196:26054 [junit4:junit4] 2> 3593 T132 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost/127.0.0.1:51336, session= id =3D 0x13aa12797c40003, negotiated timeout =3D 10000 [junit4:junit4] 2> 3594 T133 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@63c9982e name:ZooKeeperConnectio= n Watcher:127.0.0.1:51336/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 3594 T103 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 3595 T103 oascc.ZkStateReader.createClusterStateWatche= rsAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 3600 T103 oasc.ChaosMonkey.monkeyLog monkey: init - ex= pire sessions:true cause connection loss:true [junit4:junit4] 2> 3714 T103 oejs.Server.doStart jetty-8.1.2.v20120308 [junit4:junit4] 2> 3717 T103 oejs.AbstractConnector.doStart Started Selec= tChannelConnector@0.0.0.0:51150 [junit4:junit4] 2> 3718 T103 oasc.SolrResourceLoader.locateSolrHome JNDI = not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 3719 T103 oasc.SolrResourceLoader.locateSolrHome using= system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-jetty1-1351323460813 [junit4:junit4] 2> 3719 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/L= ucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.sol= r.cloud.RecoveryZkTest-jetty1-1351323460813/' [junit4:junit4] 2> 3746 T103 oass.SolrDispatchFilter.init SolrDispatchFil= ter.init() [junit4:junit4] 2> 3747 T103 oasc.SolrResourceLoader.locateSolrHome JNDI = not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 3747 T103 oasc.SolrResourceLoader.locateSolrHome using= system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-jetty1-1351323460813 [junit4:junit4] 2> 3748 T103 oasc.CoreContainer$Initializer.initialize lo= oking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nig= htlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Recove= ryZkTest-jetty1-1351323460813/solr.xml [junit4:junit4] 2> 3748 T103 oasc.CoreContainer. New CoreContainer = 1673129868 [junit4:junit4] 2> 3748 T103 oasc.CoreContainer.load Loading CoreContaine= r using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nig= htlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Recove= ryZkTest-jetty1-1351323460813/' [junit4:junit4] 2> 3749 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-So= lr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.= RecoveryZkTest-jetty1-1351323460813/' [junit4:junit4] 2> 3768 T103 oasc.CoreContainer.load Registering Log List= ener [junit4:junit4] 2> 3780 T103 oasc.CoreContainer.initZooKeeper Zookeeper c= lient=3D127.0.0.1:51336/solr [junit4:junit4] 2> 3781 T103 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:51336/solr sessionTimeout=3D5000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@e766186 [junit4:junit4] 2> 3782 T143 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:51336 [junit4:junit4] 2> 3782 T103 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 3783 T143 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost/127.0.0.1:51336, initiating sessio= n [junit4:junit4] 2> 3783 T105 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /140.211.11.196:45374 [junit4:junit4] 2> 3784 T105 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /140.211.11.196:45374 [junit4:junit4] 2> 3788 T107 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13aa12797c40004 with negotiated timeout 5000 for client /140= .211.11.196:45374 [junit4:junit4] 2> 3788 T143 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost/127.0.0.1:51336, session= id =3D 0x13aa12797c40004, negotiated timeout =3D 5000 [junit4:junit4] 2> 3789 T144 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@e766186 name:ZooKeeperConnection= Watcher:127.0.0.1:51336/solr got event WatchedEvent state:SyncConnected ty= pe:None path:null path:null type:None [junit4:junit4] 2> 3789 T103 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 3790 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40004 type:crea= te cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 3791 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40004 type:crea= te cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 3799 T103 oasc.ZkController.createEphemeralLiveNode Re= gister node as live in ZooKeeper:/live_nodes/127.0.0.1:51150_solr [junit4:junit4] 2> 3800 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40004 type:dele= te cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/live_nodes/127.0.0.1:51150_solr Error:KeeperErrorCode =3D NoNode for = /solr/live_nodes/127.0.0.1:51150_solr [junit4:junit4] 2> 3801 T103 oascc.SolrZkClient.makePath makePath: /live_= nodes/127.0.0.1:51150_solr [junit4:junit4] 2> 3804 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (1) [junit4:junit4] 2> 3805 T133 oascc.ZkStateReader$3.process Updating live = nodes... (2) [junit4:junit4] 2> 3806 T103 oashc.HttpShardHandlerFactory.getParameter S= etting socketTimeout to: 0 [junit4:junit4] 2> 3806 T103 oashc.HttpShardHandlerFactory.getParameter S= etting urlScheme to: http:// [junit4:junit4] 2> 3806 T103 oashc.HttpShardHandlerFactory.getParameter S= etting connTimeout to: 0 [junit4:junit4] 2> 3807 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 3807 T103 oashc.HttpShardHandlerFactory.getParameter S= etting corePoolSize to: 0 [junit4:junit4] 2> 3808 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 3808 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maxThreadIdleTime to: 5 [junit4:junit4] 2> 3808 T103 oashc.HttpShardHandlerFactory.getParameter S= etting sizeOfQueue to: -1 [junit4:junit4] 2> 3809 T103 oashc.HttpShardHandlerFactory.getParameter S= etting fairnessPolicy to: false [junit4:junit4] 2> 3809 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000&s= ocketTimeout=3D0&connTimeout=3D0&retry=3Dfalse [junit4:junit4] 2> 3823 T124 oascc.ZkStateReader$3.process Updating live = nodes... (2) [junit4:junit4] 2> 3828 T103 oascc.ZkStateReader.createClusterStateWatche= rsAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 3834 T103 oasc.CoreContainer.create Creating SolrCore = 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-jetty1-1351323460813/collection1 [junit4:junit4] 2> 3834 T103 oasc.ZkController.createCollectionZkNode Che= ck for collection zkNode:collection1 [junit4:junit4] 2> 3835 T103 oasc.ZkController.createCollectionZkNode Col= lection zkNode exists [junit4:junit4] 2> 3836 T103 oasc.ZkController.readConfigName Load collec= tion config from:/collections/collection1 [junit4:junit4] 2> 3838 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-So= lr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.= RecoveryZkTest-jetty1-1351323460813/collection1/' [junit4:junit4] 2> 3839 T103 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTest= s-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-j= etty1-1351323460813/collection1/lib/classes/' to classloader [junit4:junit4] 2> 3839 T103 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTest= s-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-j= etty1-1351323460813/collection1/lib/README' to classloader [junit4:junit4] 2> 3886 T103 oasc.SolrConfig. Using Lucene MatchVer= sion: LUCENE_50 [junit4:junit4] 2> 3984 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 3985 T125 oasc.Overseer$ClusterStateUpdater.updateStat= e Update state numShards=3Dnull message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":null, [junit4:junit4] 2> =09 "shard":"control_shard", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"control_collection", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:60207_solr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:60207/solr"} [junit4:junit4] 2> 4001 T103 oasc.SolrConfig. Loaded SolrConfig: so= lrconfig.xml [junit4:junit4] 2> 4021 T144 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 4021 T133 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 4021 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 4021 T103 oass.IndexSchema.readSchema Reading Solr Sch= ema [junit4:junit4] 2> 4043 T103 oass.IndexSchema.readSchema Schema name=3Dte= st [junit4:junit4] 2> 4725 T103 oass.OpenExchangeRatesOrgProvider.init Initi= alized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 4733 T103 oass.IndexSchema.readSchema default search f= ield in schema is text [junit4:junit4] 2> 4735 T103 oass.IndexSchema.readSchema unique key field= : id [junit4:junit4] 2> 4756 T103 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 4760 T103 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 4765 T103 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 4768 T103 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 4771 T103 oasc.SolrCore. [collection1] Opening n= ew SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyT= ests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkT= est-jetty1-1351323460813/collection1/, dataDir=3D./org.apache.solr.cloud.Re= coveryZkTest-1351323457216/jetty1/ [junit4:junit4] 2> 4771 T103 oasc.JmxMonitoredMap. JMX monitoring i= s enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBe= anServer@56278e83 [junit4:junit4] 2> 4772 T103 oasc.SolrCore.initDirectoryFactory org.apach= e.solr.core.MockDirectoryFactory [junit4:junit4] 2> 4772 T103 oasc.CachingDirectoryFactory.injectLockFacto= ry WARNING No lockType configured for MockDirWrapper(org.apache.lucene.stor= e.RAMDirectory@1a6313e1 lockFactory=3Dorg.apache.lucene.store.SingleInstanc= eLockFactory@5eae264f) assuming 'simple' [junit4:junit4] 2> 4773 T103 oasc.CachingDirectoryFactory.get return new = directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTe= st-1351323457216/jetty1 forceNew:false [junit4:junit4] 2> 4773 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/jetty1 [junit4:junit4] 2> 4774 T103 oasc.SolrCore.getNewIndexDir New index direc= tory detected: old=3Dnull new=3D./org.apache.solr.cloud.RecoveryZkTest-1351= 323457216/jetty1/index/ [junit4:junit4] 2> 4774 T103 oasc.SolrCore.initIndex WARNING [collection1= ] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-135132345721= 6/jetty1/index' doesn't exist. Creating new index... [junit4:junit4] 2> 4775 T103 oasc.CachingDirectoryFactory.get return new = directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTe= st-1351323457216/jetty1/index forceNew:false [junit4:junit4] 2> 4777 T103 oasc.SolrDeletionPolicy.onCommit SolrDeletio= nPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3DBaseDirectoryWrapper(org.apache.luc= ene.store.RAMDirectory@d5f0688 lockFactory=3Dorg.apache.lucene.store.Native= FSLockFactory@156237e0),segFN=3Dsegments_1,generation=3D1,filenames=3D[segm= ents_1] [junit4:junit4] 2> 4778 T103 oasc.SolrDeletionPolicy.updateCommits newest= commit =3D 1[segments_1] [junit4:junit4] 2> 4778 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/jetty1/index [junit4:junit4] 2> 4779 T103 oasc.SolrCore.initWriters created xml: solr.= XMLResponseWriter [junit4:junit4] 2> 4781 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "de= dupe" [junit4:junit4] 2> 4782 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "de= dupe-allfields" [junit4:junit4] 2> 4782 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "st= ored_sig" [junit4:junit4] 2> 4783 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "un= iq-fields" [junit4:junit4] 2> 4784 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "di= strib-dup-test-chain-implicit" [junit4:junit4] 2> 4784 T103 oasc.RequestHandlers.initHandlersFromConfig = adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 4785 T103 oasc.RequestHandlers.initHandlersFromConfig = created /replication: solr.ReplicationHandler [junit4:junit4] 2> 4785 T103 oasc.RequestHandlers.initHandlersFromConfig = created standard: solr.StandardRequestHandler [junit4:junit4] 2> 4786 T103 oasc.RequestHandlers.initHandlersFromConfig = created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 4787 T103 oasc.RequestHandlers.initHandlersFromConfig = created dismax: solr.SearchHandler [junit4:junit4] 2> 4788 T103 oasc.RequestHandlers.initHandlersFromConfig = created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler [junit4:junit4] 2> 4788 T103 oasc.RequestHandlers.initHandlersFromConfig = created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 4789 T103 oasc.RequestHandlers.initHandlersFromConfig = created defaults: solr.StandardRequestHandler [junit4:junit4] 2> 4789 T103 oasc.RequestHandlers.initHandlersFromConfig = adding lazy requestHandler: solr.StandardRequestHandler [junit4:junit4] 2> 4790 T103 oasc.RequestHandlers.initHandlersFromConfig = created lazy: solr.StandardRequestHandler [junit4:junit4] 2> 4791 T103 oasc.RequestHandlers.initHandlersFromConfig = created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 4792 T103 oasc.RequestHandlers.initHandlersFromConfig = created /terms: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 4792 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 4793 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHa= ndler [junit4:junit4] 2> 4794 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHa= ndler [junit4:junit4] 2> 4794 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.S= earchHandler [junit4:junit4] 2> 4795 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 4796 T103 oasc.RequestHandlers.initHandlersFromConfig = created tvrh: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 4797 T103 oasc.RequestHandlers.initHandlersFromConfig = created /mlt: solr.MoreLikeThisHandler [junit4:junit4] 2> 4797 T103 oasc.RequestHandlers.initHandlersFromConfig = created /debug/dump: solr.DumpRequestHandler [junit4:junit4] 2> 4803 T103 oashl.XMLLoader.init xsltCacheLifetimeSecond= s=3D60 [junit4:junit4] 2> 4807 T103 oasc.SolrCore.initDeprecatedSupport WARNING = solrconfig.xml uses deprecated , Please update your co= nfig to use the ShowFileRequestHandler. [junit4:junit4] 2> 4809 T103 oasc.SolrCore.initDeprecatedSupport WARNING = adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.= TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, = CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT] [junit4:junit4] 2> 4811 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/jetty1 [junit4:junit4] 2> 4812 T103 oass.SolrIndexSearcher. Opening Search= er@2d2c1472 main [junit4:junit4] 2> 4813 T103 oasu.CommitTracker. Hard AutoCommit: d= isabled [junit4:junit4] 2> 4813 T103 oasu.CommitTracker. Soft AutoCommit: d= isabled [junit4:junit4] 2> 4814 T103 oashc.SpellCheckComponent.inform Initializin= g spell checkers [junit4:junit4] 2> 4821 T103 oass.DirectSolrSpellChecker.init init: {name= =3Ddirect,classname=3DDirectSolrSpellChecker,field=3Dlowerfilt,minQueryLeng= th=3D3} [junit4:junit4] 2> 4877 T149 oasc.SolrCore.registerSearcher [collection1]= Registered new searcher Searcher@2d2c1472 main{StandardDirectoryReader(seg= ments_1:1)} [junit4:junit4] 2> 4881 T103 oasc.ZkController.publish numShards not foun= d on descriptor - reading it from system property [junit4:junit4] 2> 5050 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 5051 T125 oasc.Overseer$ClusterStateUpdater.updateStat= e Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:51150_solr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:51150/solr"} [junit4:junit4] 2> 5052 T125 oasc.Overseer$ClusterStateUpdater.createColl= ection Create collection collection1 with numShards 1 [junit4:junit4] 2> 5072 T133 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5072 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5072 T144 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5384 T103 oasc.CoreContainer.register registering core= : collection1 [junit4:junit4] 2> 5384 T103 oasc.ZkController.register Register shard - = core:collection1 address:http://127.0.0.1:51150/solr shardId:shard1 [junit4:junit4] 2> 5385 T103 oascc.SolrZkClient.makePath makePath: /colle= ctions/collection1/leader_elect/shard1/election [junit4:junit4] 2> 5403 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40004 type:dele= te cxid:0x45 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/collections/collection1/leaders Error:KeeperErrorCode =3D NoNode for= /solr/collections/collection1/leaders [junit4:junit4] 2> 5404 T103 oasc.ShardLeaderElectionContext.runLeaderPro= cess Running the leader process. [junit4:junit4] 2> 5405 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40004 type:crea= te cxid:0x46 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 5416 T103 oasc.ShardLeaderElectionContext.waitForRepli= casToComeUp Enough replicas found to continue. [junit4:junit4] 2> 5416 T103 oasc.ShardLeaderElectionContext.runLeaderPro= cess I may be the new leader - try and sync [junit4:junit4] 2> 5416 T103 oasc.SyncStrategy.sync Sync replicas to http= ://127.0.0.1:51150/solr/collection1/ [junit4:junit4] 2> 5417 T103 oasc.SyncStrategy.syncReplicas Sync Success = - now sync replicas to me [junit4:junit4] 2> 5417 T103 oasc.SyncStrategy.syncToMe http://127.0.0.1:= 51150/solr/collection1/ has no replicas [junit4:junit4] 2> 5417 T103 oasc.ShardLeaderElectionContext.runLeaderPro= cess I am the new leader: http://127.0.0.1:51150/solr/collection1/ [junit4:junit4] 2> 5418 T103 oascc.SolrZkClient.makePath makePath: /colle= ctions/collection1/leaders/shard1 [junit4:junit4] 2> 5423 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40004 type:crea= te cxid:0x50 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 5579 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 5596 T133 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5597 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5597 T144 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5650 T103 oasc.ZkController.register We are http://127= .0.0.1:51150/solr/collection1/ and leader is http://127.0.0.1:51150/solr/co= llection1/ [junit4:junit4] 2> 5650 T103 oasc.ZkController.register No LogReplay need= ed for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:51150/solr [junit4:junit4] 2> 5651 T103 oasc.ZkController.checkRecovery I am the lea= der, no recovery necessary [junit4:junit4] 2> 5651 T103 oasc.ZkController.publish numShards not foun= d on descriptor - reading it from system property [junit4:junit4] 2> 5655 T103 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 5656 T103 oass.SolrDispatchFilter.init user.dir=3D/usr= /home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/bui= ld/solr-core/test/J1 [junit4:junit4] 2> 5657 T103 oass.SolrDispatchFilter.init SolrDispatchFil= ter.init() done [junit4:junit4] 2> 5657 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&fol= lowRedirects=3Dfalse [junit4:junit4] 2> 5838 T103 oejs.Server.doStart jetty-8.1.2.v20120308 [junit4:junit4] 2> 5841 T103 oejs.AbstractConnector.doStart Started Selec= tChannelConnector@0.0.0.0:27739 [junit4:junit4] 2> 5842 T103 oasc.SolrResourceLoader.locateSolrHome JNDI = not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 5843 T103 oasc.SolrResourceLoader.locateSolrHome using= system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-jetty2-1351323462886 [junit4:junit4] 2> 5843 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/L= ucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.sol= r.cloud.RecoveryZkTest-jetty2-1351323462886/' [junit4:junit4] 2> 5880 T103 oass.SolrDispatchFilter.init SolrDispatchFil= ter.init() [junit4:junit4] 2> 5880 T103 oasc.SolrResourceLoader.locateSolrHome JNDI = not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 5881 T103 oasc.SolrResourceLoader.locateSolrHome using= system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-jetty2-1351323462886 [junit4:junit4] 2> 5881 T103 oasc.CoreContainer$Initializer.initialize lo= oking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nig= htlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Recove= ryZkTest-jetty2-1351323462886/solr.xml [junit4:junit4] 2> 5882 T103 oasc.CoreContainer. New CoreContainer = 138935409 [junit4:junit4] 2> 5883 T103 oasc.CoreContainer.load Loading CoreContaine= r using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Nig= htlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Recove= ryZkTest-jetty2-1351323462886/' [junit4:junit4] 2> 5883 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-So= lr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.= RecoveryZkTest-jetty2-1351323462886/' [junit4:junit4] 2> 5909 T103 oasc.CoreContainer.load Registering Log List= ener [junit4:junit4] 2> 5924 T103 oasc.CoreContainer.initZooKeeper Zookeeper c= lient=3D127.0.0.1:51336/solr [junit4:junit4] 2> 5925 T103 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:51336/solr sessionTimeout=3D5000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@32831c19 [junit4:junit4] 2> 5926 T159 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:51336 [junit4:junit4] 2> 5926 T103 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 5928 T159 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost/127.0.0.1:51336, initiating sessio= n [junit4:junit4] 2> 5928 T105 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /140.211.11.196:15632 [junit4:junit4] 2> 5929 T105 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /140.211.11.196:15632 [junit4:junit4] 2> 5931 T107 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13aa12797c40005 with negotiated timeout 5000 for client /140= .211.11.196:15632 [junit4:junit4] 2> 5932 T159 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost/127.0.0.1:51336, session= id =3D 0x13aa12797c40005, negotiated timeout =3D 5000 [junit4:junit4] 2> 5932 T160 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@32831c19 name:ZooKeeperConnectio= n Watcher:127.0.0.1:51336/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 5933 T103 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 5934 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40005 type:crea= te cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 5947 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40005 type:crea= te cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 5949 T103 oasc.ZkController.createEphemeralLiveNode Re= gister node as live in ZooKeeper:/live_nodes/127.0.0.1:27739_solr [junit4:junit4] 2> 5950 T108 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13aa12797c40005 type:dele= te cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/live_nodes/127.0.0.1:27739_solr Error:KeeperErrorCode =3D NoNode for = /solr/live_nodes/127.0.0.1:27739_solr [junit4:junit4] 2> 5952 T103 oascc.SolrZkClient.makePath makePath: /live_= nodes/127.0.0.1:27739_solr [junit4:junit4] 2> 5955 T144 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5955 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (2) [junit4:junit4] 2> 5956 T133 oascc.ZkStateReader$3.process Updating live = nodes... (3) [junit4:junit4] 2> 5957 T133 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 5957 T103 oashc.HttpShardHandlerFactory.getParameter S= etting socketTimeout to: 0 [junit4:junit4] 2> 5958 T124 oascc.ZkStateReader$3.process Updating live = nodes... (3) [junit4:junit4] 2> 5958 T144 oascc.ZkStateReader$3.process Updating live = nodes... (3) [junit4:junit4] 2> 5958 T103 oashc.HttpShardHandlerFactory.getParameter S= etting urlScheme to: http:// [junit4:junit4] 2> 5959 T103 oashc.HttpShardHandlerFactory.getParameter S= etting connTimeout to: 0 [junit4:junit4] 2> 5959 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 5960 T103 oashc.HttpShardHandlerFactory.getParameter S= etting corePoolSize to: 0 [junit4:junit4] 2> 5961 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 5961 T103 oashc.HttpShardHandlerFactory.getParameter S= etting maxThreadIdleTime to: 5 [junit4:junit4] 2> 5962 T103 oashc.HttpShardHandlerFactory.getParameter S= etting sizeOfQueue to: -1 [junit4:junit4] 2> 5962 T103 oashc.HttpShardHandlerFactory.getParameter S= etting fairnessPolicy to: false [junit4:junit4] 2> 5963 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000&s= ocketTimeout=3D0&connTimeout=3D0&retry=3Dfalse [junit4:junit4] 2> 5982 T103 oascc.ZkStateReader.createClusterStateWatche= rsAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 5990 T103 oasc.CoreContainer.create Creating SolrCore = 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lu= cene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr= .cloud.RecoveryZkTest-jetty2-1351323462886/collection1 [junit4:junit4] 2> 5990 T103 oasc.ZkController.createCollectionZkNode Che= ck for collection zkNode:collection1 [junit4:junit4] 2> 5991 T103 oasc.ZkController.createCollectionZkNode Col= lection zkNode exists [junit4:junit4] 2> 5992 T103 oasc.ZkController.readConfigName Load collec= tion config from:/collections/collection1 [junit4:junit4] 2> 5994 T103 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-So= lr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.= RecoveryZkTest-jetty2-1351323462886/collection1/' [junit4:junit4] 2> 5995 T103 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTest= s-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-j= etty2-1351323462886/collection1/lib/README' to classloader [junit4:junit4] 2> 5996 T103 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTest= s-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-j= etty2-1351323462886/collection1/lib/classes/' to classloader [junit4:junit4] 2> 6053 T103 oasc.SolrConfig. Using Lucene MatchVer= sion: LUCENE_50 [junit4:junit4] 2> 6111 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 6112 T125 oasc.Overseer$ClusterStateUpdater.updateStat= e Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "shard":"shard1", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:51150_solr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:51150/solr"} [junit4:junit4] 2> 6122 T103 oasc.SolrConfig. Loaded SolrConfig: so= lrconfig.xml [junit4:junit4] 2> 6149 T133 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 6149 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 6149 T144 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 6235 T160 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 6235 T103 oass.IndexSchema.readSchema Reading Solr Sch= ema [junit4:junit4] 2> 6245 T103 oass.IndexSchema.readSchema Schema name=3Dte= st [junit4:junit4] 2> 6856 T103 oass.OpenExchangeRatesOrgProvider.init Initi= alized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 6869 T103 oass.IndexSchema.readSchema default search f= ield in schema is text [junit4:junit4] 2> 6873 T103 oass.IndexSchema.readSchema unique key field= : id [junit4:junit4] 2> 6896 T103 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 6902 T103 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 6907 T103 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 6911 T103 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 6914 T103 oasc.SolrCore. [collection1] Opening n= ew SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyT= ests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkT= est-jetty2-1351323462886/collection1/, dataDir=3D./org.apache.solr.cloud.Re= coveryZkTest-1351323457216/jetty2/ [junit4:junit4] 2> 6915 T103 oasc.JmxMonitoredMap. JMX monitoring i= s enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBe= anServer@56278e83 [junit4:junit4] 2> 6916 T103 oasc.SolrCore.initDirectoryFactory org.apach= e.solr.core.MockDirectoryFactory [junit4:junit4] 2> 6917 T103 oasc.CachingDirectoryFactory.injectLockFacto= ry WARNING No lockType configured for MockDirWrapper(org.apache.lucene.stor= e.RAMDirectory@7c3f6843 lockFactory=3Dorg.apache.lucene.store.SingleInstanc= eLockFactory@26d6466f) assuming 'simple' [junit4:junit4] 2> 6917 T103 oasc.CachingDirectoryFactory.get return new = directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTe= st-1351323457216/jetty2 forceNew:false [junit4:junit4] 2> 6918 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/jetty2 [junit4:junit4] 2> 6918 T103 oasc.SolrCore.getNewIndexDir New index direc= tory detected: old=3Dnull new=3D./org.apache.solr.cloud.RecoveryZkTest-1351= 323457216/jetty2/index/ [junit4:junit4] 2> 6919 T103 oasc.SolrCore.initIndex WARNING [collection1= ] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-135132345721= 6/jetty2/index' doesn't exist. Creating new index... [junit4:junit4] 2> 6920 T103 oasc.CachingDirectoryFactory.get return new = directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTe= sts-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTe= st-1351323457216/jetty2/index forceNew:false [junit4:junit4] 2> 6924 T103 oasc.SolrDeletionPolicy.onCommit SolrDeletio= nPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3DMockDirWrapper(org.apache.lucene.st= ore.RAMDirectory@79a5b7b4 lockFactory=3Dorg.apache.lucene.store.NativeFSLoc= kFactory@7119087d),segFN=3Dsegments_1,generation=3D1,filenames=3D[segments_= 1] [junit4:junit4] 2> 6924 T103 oasc.SolrDeletionPolicy.updateCommits newest= commit =3D 1[segments_1] [junit4:junit4] 2> 6925 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/jetty2/index [junit4:junit4] 2> 6926 T103 oasc.SolrCore.initWriters created xml: solr.= XMLResponseWriter [junit4:junit4] 2> 6929 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "de= dupe" [junit4:junit4] 2> 6930 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "de= dupe-allfields" [junit4:junit4] 2> 6930 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "st= ored_sig" [junit4:junit4] 2> 6931 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "un= iq-fields" [junit4:junit4] 2> 6932 T103 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "di= strib-dup-test-chain-implicit" [junit4:junit4] 2> 6933 T103 oasc.RequestHandlers.initHandlersFromConfig = adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 6934 T103 oasc.RequestHandlers.initHandlersFromConfig = created /replication: solr.ReplicationHandler [junit4:junit4] 2> 6935 T103 oasc.RequestHandlers.initHandlersFromConfig = created standard: solr.StandardRequestHandler [junit4:junit4] 2> 6936 T103 oasc.RequestHandlers.initHandlersFromConfig = created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 6937 T103 oasc.RequestHandlers.initHandlersFromConfig = created dismax: solr.SearchHandler [junit4:junit4] 2> 6938 T103 oasc.RequestHandlers.initHandlersFromConfig = created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler [junit4:junit4] 2> 6939 T103 oasc.RequestHandlers.initHandlersFromConfig = created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 6940 T103 oasc.RequestHandlers.initHandlersFromConfig = created defaults: solr.StandardRequestHandler [junit4:junit4] 2> 6941 T103 oasc.RequestHandlers.initHandlersFromConfig = adding lazy requestHandler: solr.StandardRequestHandler [junit4:junit4] 2> 6942 T103 oasc.RequestHandlers.initHandlersFromConfig = created lazy: solr.StandardRequestHandler [junit4:junit4] 2> 6943 T103 oasc.RequestHandlers.initHandlersFromConfig = created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 6944 T103 oasc.RequestHandlers.initHandlersFromConfig = created /terms: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 6945 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 6946 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHa= ndler [junit4:junit4] 2> 6947 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHa= ndler [junit4:junit4] 2> 6948 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.S= earchHandler [junit4:junit4] 2> 6950 T103 oasc.RequestHandlers.initHandlersFromConfig = created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 6951 T103 oasc.RequestHandlers.initHandlersFromConfig = created tvrh: org.apache.solr.handler.component.SearchHandler [junit4:junit4] 2> 6952 T103 oasc.RequestHandlers.initHandlersFromConfig = created /mlt: solr.MoreLikeThisHandler [junit4:junit4] 2> 6953 T103 oasc.RequestHandlers.initHandlersFromConfig = created /debug/dump: solr.DumpRequestHandler [junit4:junit4] 2> 6961 T103 oashl.XMLLoader.init xsltCacheLifetimeSecond= s=3D60 [junit4:junit4] 2> 6966 T103 oasc.SolrCore.initDeprecatedSupport WARNING = solrconfig.xml uses deprecated , Please update your co= nfig to use the ShowFileRequestHandler. [junit4:junit4] 2> 6968 T103 oasc.SolrCore.initDeprecatedSupport WARNING = adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.= TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, = CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT] [junit4:junit4] 2> 6970 T103 oasc.CachingDirectoryFactory.close Releasing= directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests= -trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-= 1351323457216/jetty2 [junit4:junit4] 2> 6971 T103 oass.SolrIndexSearcher. Opening Search= er@532c734c main [junit4:junit4] 2> 6973 T103 oasu.CommitTracker. Hard AutoCommit: d= isabled [junit4:junit4] 2> 6973 T103 oasu.CommitTracker. Soft AutoCommit: d= isabled [junit4:junit4] 2> 6975 T103 oashc.SpellCheckComponent.inform Initializin= g spell checkers [junit4:junit4] 2> 6984 T103 oass.DirectSolrSpellChecker.init init: {name= =3Ddirect,classname=3DDirectSolrSpellChecker,field=3Dlowerfilt,minQueryLeng= th=3D3} [junit4:junit4] 2> 7055 T165 oasc.SolrCore.registerSearcher [collection1]= Registered new searcher Searcher@532c734c main{StandardDirectoryReader(seg= ments_1:1)} [junit4:junit4] 2> 7066 T103 oasc.ZkController.publish numShards not foun= d on descriptor - reading it from system property [junit4:junit4] 2> 7158 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 7160 T125 oasc.Overseer$ClusterStateUpdater.updateStat= e Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:27739_solr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:27739/solr"} [junit4:junit4] 2> 7176 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 7176 T160 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 7176 T144 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 7176 T133 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 7569 T103 oasc.CoreContainer.register registering core= : collection1 [junit4:junit4] 2> 7569 T103 oasc.ZkController.register Register shard - = core:collection1 address:http://127.0.0.1:27739/solr shardId:shard1 [junit4:junit4] 2> 7575 T103 oasc.ZkController.register We are http://127= .0.0.1:27739/solr/collection1/ and leader is http://127.0.0.1:51150/solr/co= llection1/ [junit4:junit4] 2> 7576 T103 oasc.ZkController.register No LogReplay need= ed for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:27739/solr [junit4:junit4] 2> 7577 T103 oasc.ZkController.checkRecovery Core needs t= o recover:collection1 [junit4:junit4] 2> 7578 T103 oasu.DefaultSolrCoreState.doRecovery Running= recovery - first canceling any ongoing recovery [junit4:junit4] 2> ASYNC NEW_CORE C82 name=3Dcollection1 org.apache.solr= .core.SolrCore@4d29efd0 url=3Dhttp://127.0.0.1:27739/solr/collection1 node= =3D127.0.0.1:27739_solr [junit4:junit4] 2> 7589 T166 C82 P27739 oasc.RecoveryStrategy.run Startin= g recovery process. core=3Dcollection1 recoveringAfterStartup=3Dtrue [junit4:junit4] 2> 7589 T166 C82 P27739 oasc.RecoveryStrategy.doRecovery = ###### startupVersions=3D[] [junit4:junit4] 2> 7589 T103 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 7590 T166 C82 P27739 oasc.ZkController.publish numShar= ds not found on descriptor - reading it from system property [junit4:junit4] 2> 7592 T103 oass.SolrDispatchFilter.init user.dir=3D/usr= /home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/bui= ld/solr-core/test/J1 [junit4:junit4] 2> 7593 T103 oass.SolrDispatchFilter.init SolrDispatchFil= ter.init() done [junit4:junit4] 2> 7593 T166 C82 P27739 oasc.RecoveryStrategy.doRecovery = Attempting to PeerSync from http://127.0.0.1:51150/solr/collection1/ core= =3Dcollection1 - recoveringAfterStartup=3Dtrue [junit4:junit4] 2> 7594 T103 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&fol= lowRedirects=3Dfalse [junit4:junit4] 2> 7605 T166 C82 P27739 oascsi.HttpClientUtil.createClien= t Creating new http client, config:maxConnectionsPerHost=3D20&maxConnection= s=3D10000&connTimeout=3D30000&socketTimeout=3D30000&retry=3Dfalse [junit4:junit4] 2> 7610 T103 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 7683 T125 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 7684 T125 oasc.Overseer$ClusterStateUpdater.updateStat= e Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "shard":"shard1", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:27739_solr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:27739/solr"} [junit4:junit4] 2> 7698 T124 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 7698 T160 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 7698 T133 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> 7698 T144 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (3) [junit4:junit4] 2> C82_STATE=3Dcoll:collection1 core:collection1 props:{= shard=3Dshard1, roles=3Dnull, state=3Drecovering, core=3Dcollection1, colle= ction=3Dcollection1, node_name=3D127.0.0.1:27739_solr, base_url=3Dhttp://12= 7.0.0.1:27739/solr} [junit4:junit4] 2> 7716 T166 C82 P27739 oasu.PeerSync.sync PeerSync: core= =3Dcollection1 url=3Dhttp://127.0.0.1:27739/solr START replicas=3D[http://1= 27.0.0.1:51150/solr/collection1/] nUpdates=3D100 [junit4:junit4] 2> 7738 T166 C82 P27739 oasu.PeerSync.sync WARNING no fra= me of reference to tell of we've missed updates [junit4:junit4] 2> 7739 T166 C82 P27739 oasc.RecoveryStrategy.doRecovery = PeerSync Recovery was not successful - trying replication. core=3Dcollectio= n1 [junit4:junit4] 2> 7739 T166 C82 P27739 oasc.RecoveryStrategy.doRecovery = Starting Replication Recovery. core=3Dcollection1 [junit4:junit4] 2> 7739 T166 C82 P27739 oascsi.HttpClientUtil.createClien= t Creating new http client, config:maxConnections=3D128&maxConnectionsPerHo= st=3D32&followRedirects=3Dfalse [junit4:junit4] 2> ASYNC NEW_CORE C83 name=3Dcollection1 org.apache.solr= .core.SolrCore@7f0abe88 url=3Dhttp://127.0.0.1:51150/solr/collection1 node= =3D127.0.0.1:51150_solr [junit4:junit4] 2> 9221 T146 C83 P51150 REQ /get {distrib=3Dfalse&qt=3D/g= et&wt=3Djavabin&version=3D2&getVersions=3D100} status=3D0 QTime=3D12=20 [junit4:junit4] 2> ASYNC NEW_CORE C84 name=3Dcollection1 org.apache.solr= .core.SolrCore@52fa7139 url=3Dhttp://127.0.0.1:60207/solr/collection1 node= =3D127.0.0.1:60207_solr [junit4:junit4] 2> 9348 T127 C84 P60207 oascsi.HttpClientUtil.createClien= t Creating new http client, config:maxConnections=3D500&maxConnectionsPerHo= st=3D16 [junit4:junit4] 2> 9414 T127 C84 P60207 oasc.CachingDirectoryFactory.clos= e Releasing directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-N= ightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Reco= veryZkTest-1351323457216/control/data [junit4:junit4] 2> 9418 T127 C84 P60207 oasc.SolrDeletionPolicy.onInit So= lrDeletionPolicy.onInit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3DMockDirWrapper(org.apache.lucene.st= ore.RAMDirectory@d9438de lockFactory=3Dorg.apache.lucene.store.NativeFSLock= Factory@48c91629),segFN=3Dsegments_1,generation=3D1,filenames=3D[segments_1= ] [junit4:junit4] 2> 9419 T127 C84 P60207 oasc.SolrDeletionPolicy.updateCom= mits newest commit =3D 1[segments_1] [junit4:junit4] 2> 9448 T129 C84 P60207 /update {wt=3Djavabin&version=3D2= } {add=3D[10001 (1416965355340824577)]} 0 179 [junit4:junit4] 2> 9449 T127 C84 P60207 /update {wt=3Djavabin&version=3D2= } {add=3D[1 (1416965355340824576)]} 0 180 [junit4:junit4] 2> 9474 T148 C83 P51150 oasc.CachingDirectoryFactory.clos= e Releasing directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-N= ightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.Reco= veryZkTest-1351323457216/jetty1 [junit4:junit4] 2> 9476 T148 C83 P51150 oasc.SolrDeletionPolicy.onInit So= lrDeletionPolicy.onInit: commits: [...truncated too long message...] server.AbstractConnector$Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 11) Thread[id=3D193, name=3Dqtp643552837-193 Accepto= r2 SelectChannelConnector@0.0.0.0:36458, state=3DRUNNABLE, group=3DTGRP-Sol= rCmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept0(= Native Method) [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:163) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 12) Thread[id=3D207, name=3Dqtp1236323694-207 Accept= or3 SelectChannelConnector@0.0.0.0:31983, state=3DBLOCKED, group=3DTGRP-Sol= rCmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:147) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 13) Thread[id=3D217, name=3Dqtp144414042-217 Accepto= r1 SelectChannelConnector@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-Solr= CmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:147) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 14) Thread[id=3D229, name=3Dqtp837657752-229 Accepto= r0 SelectChannelConnector@0.0.0.0:53656, state=3DBLOCKED, group=3DTGRP-Solr= CmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:147) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 15) Thread[id=3D218, name=3Dqtp144414042-218 Accepto= r2 SelectChannelConnector@0.0.0.0:48493, state=3DRUNNABLE, group=3DTGRP-Sol= rCmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept0(= Native Method) [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:163) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 16) Thread[id=3D245, name=3Dqtp149836331-245 Accepto= r2 SelectChannelConnector@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-Solr= CmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:147) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 17) Thread[id=3D243, name=3Dqtp149836331-243 Accepto= r0 SelectChannelConnector@0.0.0.0:16444, state=3DRUNNABLE, group=3DTGRP-Sol= rCmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept0(= Native Method) [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:163) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 18) Thread[id=3D194, name=3Dqtp643552837-194 Accepto= r3 SelectChannelConnector@0.0.0.0:36458, state=3DBLOCKED, group=3DTGRP-Solr= CmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:147) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 19) Thread[id=3D246, name=3Dqtp149836331-246 Accepto= r3 SelectChannelConnector@0.0.0.0:16444, state=3DBLOCKED, group=3DTGRP-Solr= CmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:147) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > 20) Thread[id=3D219, name=3Dqtp144414042-219 Accepto= r3 SelectChannelConnector@0.0.0.0:48493, state=3DBLOCKED, group=3DTGRP-Solr= CmdDistributorTest] [junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(S= erverSocketChannelImpl.java:147) [junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelC= onnector.accept(SelectChannelConnector.java:104) [junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$= Acceptor.run(AbstractConnector.java:933) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:599) [junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:534) [junit4:junit4] > at java.lang.Thread.run(Thread.java:679) [junit4:junit4] > =09at __randomizedtesting.SeedInfo.seed([A6F29E4E81D42= F0D]:0) [junit4:junit4] Completed on J0 in 33.16s, 1 test, 2 errors <<< FAILURES! [...truncated 56 lines...] BUILD FAILED /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/buil= d.xml:335: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/buil= d.xml:39: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr= /build.xml:178: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/luce= ne/module-build.xml:61: The following error occurred while executing this l= ine: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/luce= ne/common-build.xml:1159: The following error occurred while executing this= line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/luce= ne/common-build.xml:827: There were test failures: 232 suites, 961 tests, 4= suite-level errors, 691 ignored Total time: 65 minutes 9 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Email was triggered for: Failure Sending email for trigger: Failure ------=_Part_559_14101358.1351323582978 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org ------=_Part_559_14101358.1351323582978--