Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D9FB3200B74 for ; Thu, 1 Sep 2016 11:59:53 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id D8451160AB5; Thu, 1 Sep 2016 09:59:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6042B160AAE for ; Thu, 1 Sep 2016 11:59:51 +0200 (CEST) Received: (qmail 15883 invoked by uid 500); 1 Sep 2016 09:59:45 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 15865 invoked by uid 99); 1 Sep 2016 09:59:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Sep 2016 09:59:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 652131A00FD for ; Thu, 1 Sep 2016 09:59:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[SPF_PASS=-0.001] autolearn=disabled Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id rMYzTBTJt95n for ; Thu, 1 Sep 2016 09:59:33 +0000 (UTC) Received: from serv1.sd-datasolutions.de (serv1.sd-datasolutions.de [188.138.57.78]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id B67535F233 for ; Thu, 1 Sep 2016 09:59:32 +0000 (UTC) Received: from serv1 (localhost.localdomain [127.0.0.1]) by serv1.sd-datasolutions.de (Postfix) with ESMTP id 4A6354C00E2; Thu, 1 Sep 2016 09:59:32 +0000 (UTC) Date: Thu, 1 Sep 2016 09:57:52 +0000 (UTC) From: Policeman Jenkins Server To: dsmiley@apache.org, dev@lucene.apache.org Message-ID: <141103265.103.1472723972306.JavaMail.jenkins@serv1> Subject: [JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 822 - Unstable! MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_102_1908515820.1472723972266" X-Jenkins-Job: Lucene-Solr-master-Solaris X-Jenkins-Result: UNSTABLE archived-at: Thu, 01 Sep 2016 09:59:54 -0000 ------=_Part_102_1908515820.1472723972266 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/822/ Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC 4 tests failed. FAILED: org.apache.solr.cloud.BasicDistributedZk2Test.test Error Message: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0= .0.1:39555 within 30000 ms Stack Trace: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException= : Could not connect to ZooKeeper 127.0.0.1:39555 within 30000 ms =09at __randomizedtesting.SeedInfo.seed([6E953035E71A70BF:E6C10FEF49E61D47]= :0) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:18= 1) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:11= 5) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:11= 0) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:97= ) =09at org.apache.solr.cloud.AbstractDistribZkTestBase.printLayout(AbstractD= istribZkTestBase.java:295) =09at org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribTearDown(A= bstractFullDistribZkTestBase.java:1500) =09at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$Shards= Statement.evaluate(BaseDistributedSearchTestCase.java:962) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRu= leSetupTeardownChained.java:49) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleT= hreadAndTestName.java:48) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTa= sk(ThreadLeakControl.java:809) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(Threa= dLeakControl.java:460) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran= domizedRunner.java:880) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Random= izedRunner.java:781) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random= izedRunner.java:816) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:827) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooK= eeper 127.0.0.1:39555 within 30000 ms =09at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(Conne= ctionManager.java:235) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:17= 3) =09... 37 more FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk= 2Test Error Message: 68 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistribute= dZk2Test: 1) Thread[id=3D1475, name=3DProcessThread(sid:0 cport:-1):, s= tate=3DWAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.U= nsafe.park(Native Method) at java.util.concurrent.locks.LockSupport= .park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQ= ueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:203= 9) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQ= ueue.java:442) at org.apache.zookeeper.server.PrepRequestProcessor.= run(PrepRequestProcessor.java:120) 2) Thread[id=3D1545, name=3Dorg.eclip= se.jetty.server.session.HashSessionManager@25fd8ffdTimer, state=3DTIMED_WAI= TING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park= (Native Method) at java.util.concurrent.locks.LockSupport.parkNanos= (LockSupport.java:215) at java.util.concurrent.locks.AbstractQueued= Synchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:207= 8) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQ= ueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concu= rrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPool= Executor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTa= sk(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPool= Executor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concu= rrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at= java.lang.Thread.run(Thread.java:745) 3) Thread[id=3D1494, name=3Dqtp19= 8503387-1494, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] = at sun.misc.Unsafe.park(Native Method) at java.util.concurr= ent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.= concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abst= ractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.Blockin= gArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.= util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) = at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadP= ool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.ru= n(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.java:74= 5) 4) Thread[id=3D1567, name=3Dqtp395667898-1567, state=3DTIMED_WAITING,= group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Nati= ve Method) at java.util.concurrent.locks.LockSupport.parkNanos(Lock= Support.java:215) at java.util.concurrent.locks.AbstractQueuedSynch= ronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) = at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.= java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJob= Poll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.Qu= euedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.= jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) = at java.lang.Thread.run(Thread.java:745) 5) Thread[id=3D1473, name=3DSe= ssionTracker, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] = at java.lang.Object.wait(Native Method) at org.apache.zooke= eper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) 6) Threa= d[id=3D1483, name=3Dqtp198503387-1483, state=3DTIMED_WAITING, group=3DTGRP-= BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) = at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:2= 15) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Condit= ionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.e= clipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThr= eadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool= .access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thr= ead.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.= Thread.run(Thread.java:745) 7) Thread[id=3D1516, name=3DConnection evict= or, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at= java.lang.Thread.sleep(Native Method) at org.apache.http.impl.clie= nt.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at ja= va.lang.Thread.run(Thread.java:745) 8) Thread[id=3D1472, name=3DNIOServe= rCxn.Factory:0.0.0.0/0.0.0.0:0, state=3DRUNNABLE, group=3DTGRP-BasicDistrib= utedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)= at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:22= 3) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.j= ava:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.jav= a:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) = at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFac= tory.java:178) at java.lang.Thread.run(Thread.java:745) 9) Threa= d[id=3D1526, name=3Dqtp1344769124-1526-acceptor-0@5df1cecf-ServerConnector@= 34661a88{HTTP/1.1,[http/1.1]}{127.0.0.1:39555}, state=3DRUNNABLE, group=3DT= GRP-BasicDistributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.= accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept= (ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChanne= lImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty= .server.ServerConnector.accept(ServerConnector.java:382) at org.ecl= ipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593= ) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedTh= readPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPoo= l$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.j= ava:745) 10) Thread[id=3D1572, name=3DConnection evictor, state=3DTIMED_W= AITING, group=3DTGRP-BasicDistributedZk2Test] at java.lang.Thread.s= leep(Native Method) at org.apache.http.impl.client.IdleConnectionEv= ictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(= Thread.java:745) 11) Thread[id=3D1504, name=3DOverseerStateUpdate-9651626= 3976960004-127.0.0.1:43794_-n_0000000000, state=3DTIMED_WAITING, group=3DOv= erseer state updater.] at sun.misc.Unsafe.park(Native Method) = at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215= ) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Conditio= nObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.apa= che.solr.cloud.DistributedQueue.peek(DistributedQueue.java:164) at = org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:138) = at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java= :196) at java.lang.Thread.run(Thread.java:745) 12) Thread[id=3D15= 65, name=3Dqtp395667898-1565, state=3DTIMED_WAITING, group=3DTGRP-BasicDist= ributedZk2Test] at sun.misc.Unsafe.park(Native Method) at j= ava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) = at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject= .awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.je= tty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at or= g.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.j= ava:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$8= 00(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.Queue= dThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.ru= n(Thread.java:745) 13) Thread[id=3D1487, name=3Dqtp198503387-1487, state= =3DRUNNABLE, group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.De= vPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWr= apper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelec= torImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.Selecto= rImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorI= mpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(= SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$Sele= ctorProducer.select(ManagedSelector.java:233) at org.eclipse.jetty.= io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181) = at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceA= ndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thr= ead.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:102) = at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:13= 7) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedT= hreadPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPo= ol$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.= java:745) 14) Thread[id=3D1527, name=3Dqtp1344769124-1527, state=3DTIMED_= WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.p= ark(Native Method) at java.util.concurrent.locks.LockSupport.parkNa= nos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQue= uedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:= 2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArr= ayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool= .idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.t= hread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.= eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)= at java.lang.Thread.run(Thread.java:745) 15) Thread[id=3D1536, n= ame=3DConnection evictor, state=3DTIMED_WAITING, group=3DTGRP-BasicDistribu= tedZk2Test] at java.lang.Thread.sleep(Native Method) at org= .apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.= java:66) at java.lang.Thread.run(Thread.java:745) 16) Thread[id= =3D1562, name=3Dqtp395667898-1562-acceptor-0@5bead1a8-ServerConnector@20530= 6da{HTTP/1.1,[http/1.1]}{127.0.0.1:40345}, state=3DRUNNABLE, group=3DTGRP-B= asicDistributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.accep= t0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(Serv= erSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl= .accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.serv= er.ServerConnector.accept(ServerConnector.java:382) at org.eclipse.= jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) = at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadP= ool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.r= un(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:7= 45) 17) Thread[id=3D1497, name=3DConnection evictor, state=3DTIMED_WAITIN= G, group=3DTGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(= Native Method) at org.apache.http.impl.client.IdleConnectionEvictor= $1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Threa= d.java:745) 18) Thread[id=3D1552, name=3Dqtp1050973900-1552, state=3DTIME= D_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe= .park(Native Method) at java.util.concurrent.locks.LockSupport.park= Nanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQ= ueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.jav= a:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingA= rrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPo= ol.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util= .thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at or= g.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:60= 9) at java.lang.Thread.run(Thread.java:745) 19) Thread[id=3D1515,= name=3DConnection evictor, state=3DTIMED_WAITING, group=3DTGRP-BasicDistri= butedZk2Test] at java.lang.Thread.sleep(Native Method) at o= rg.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvicto= r.java:66) at java.lang.Thread.run(Thread.java:745) 20) Thread[id= =3D1532, name=3Dqtp1344769124-1532, state=3DTIMED_WAITING, group=3DTGRP-Bas= icDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) = at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)= at java.util.concurrent.locks.AbstractQueuedSynchronizer$Condition= Object.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.ecli= pse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThread= Pool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.ac= cess$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread= .QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thr= ead.run(Thread.java:745) 21) Thread[id=3D1490, name=3Dorg.eclipse.jetty.s= erver.session.HashSessionManager@3b500f87Timer, state=3DTIMED_WAITING, grou= p=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Me= thod) at java.util.concurrent.locks.LockSupport.parkNanos(LockSuppo= rt.java:215) at java.util.concurrent.locks.AbstractQueuedSynchroniz= er$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) = at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(= ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.Sche= duledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.j= ava:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadP= oolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.r= unWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.Thre= adPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang= .Thread.run(Thread.java:745) 22) Thread[id=3D1506, name=3DOverseerHdfsCor= eFailoverThread-96516263976960004-127.0.0.1:43794_-n_0000000000, state=3DTI= MED_WAITING, group=3DOverseer Hdfs SolrCore Failover Thread.] at ja= va.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.Overse= erAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)= at java.lang.Thread.run(Thread.java:745) 23) Thread[id=3D1568, n= ame=3Dqtp395667898-1568, state=3DTIMED_WAITING, group=3DTGRP-BasicDistribut= edZk2Test] at sun.misc.Unsafe.park(Native Method) at java.u= til.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at= java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awai= tNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.u= til.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:5= 46) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Qu= euedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThre= adPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thr= ead.java:745) 24) Thread[id=3D1566, name=3Dqtp395667898-1566, state=3DTIM= ED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsaf= e.park(Native Method) at java.util.concurrent.locks.LockSupport.par= kNanos(LockSupport.java:215) at java.util.concurrent.locks.Abstract= QueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.ja= va:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(Blocking= ArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadP= ool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.uti= l.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at o= rg.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:6= 09) at java.lang.Thread.run(Thread.java:745) 25) Thread[id=3D1543= , name=3Dqtp567249146-1543, state=3DTIMED_WAITING, group=3DTGRP-BasicDistri= butedZk2Test] at sun.misc.Unsafe.park(Native Method) at jav= a.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) = at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.a= waitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jett= y.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.= eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.jav= a:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800= (QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedT= hreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(= Thread.java:745) 26) Thread[id=3D1556, name=3Dqtp1050973900-1556, state= =3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc= .Unsafe.park(Native Method) at java.util.concurrent.locks.LockSuppo= rt.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.Ab= stractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroni= zer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(Bl= ockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedT= hreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jet= ty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.= java:609) at java.lang.Thread.run(Thread.java:745) 27) Thread[id= =3D1486, name=3Dqtp198503387-1486, state=3DTIMED_WAITING, group=3DTGRP-Basi= cDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) = at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) = at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclip= se.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadP= ool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.acc= ess$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.= QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thre= ad.run(Thread.java:745) 28) Thread[id=3D1485, name=3Dqtp198503387-1485-ac= ceptor-0@69afa4fc-ServerConnector@2281f6fb{HTTP/1.1,[http/1.1]}{127.0.0.1:4= 3794}, state=3DRUNNABLE, group=3DTGRP-BasicDistributedZk2Test] at s= un.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio= .ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) = at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java= :250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConn= ector.java:382) at org.eclipse.jetty.server.AbstractConnector$Accep= tor.run(AbstractConnector.java:593) at org.eclipse.jetty.util.threa= d.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org.eclipse= .jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) = at java.lang.Thread.run(Thread.java:745) 29) Thread[id=3D1522, name=3Dq= tp1344769124-1522, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2T= est] at sun.misc.Unsafe.park(Native Method) at java.util.co= ncurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.= util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos= (AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.Bl= ockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.j= etty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) = at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedTh= readPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool= $3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread.ja= va:745) 30) Thread[id=3D1564, name=3Dqtp395667898-1564, state=3DTIMED_WAI= TING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park= (Native Method) at java.util.concurrent.locks.LockSupport.parkNanos= (LockSupport.java:215) at java.util.concurrent.locks.AbstractQueued= Synchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:207= 8) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQ= ueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.id= leJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thre= ad.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) = at java.lang.Thread.run(Thread.java:745) 31) Thread[id=3D1544, name= =3Dqtp567249146-1544, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZ= k2Test] at sun.misc.Unsafe.park(Native Method) at java.util= .concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at ja= va.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNa= nos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util= .BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclips= e.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)= at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queue= dThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadP= ool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Thread= .java:745) 32) Thread[id=3D1523, name=3Dqtp1344769124-1523, state=3DTIMED= _WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.= park(Native Method) at java.util.concurrent.locks.LockSupport.parkN= anos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQu= euedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java= :2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingAr= rayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPoo= l.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.= thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org= .eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609= ) at java.lang.Thread.run(Thread.java:745) 33) Thread[id=3D1569, = name=3Dorg.eclipse.jetty.server.session.HashSessionManager@4f550564Timer, s= tate=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.= misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockS= upport.parkNanos(LockSupport.java:215) at java.util.concurrent.lock= s.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynch= ronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecu= tor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at= java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPo= olExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concu= rrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at= java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :617) at java.lang.Thread.run(Thread.java:745) 34) Thread[id=3D15= 63, name=3Dqtp395667898-1563, state=3DTIMED_WAITING, group=3DTGRP-BasicDist= ributedZk2Test] at sun.misc.Unsafe.park(Native Method) at j= ava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) = at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject= .awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.je= tty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at or= g.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.j= ava:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$8= 00(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.Queue= dThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.ru= n(Thread.java:745) 35) Thread[id=3D1482, name=3Dqtp198503387-1482, state= =3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc= .Unsafe.park(Native Method) at java.util.concurrent.locks.LockSuppo= rt.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.Ab= stractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroni= zer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(Bl= ockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedT= hreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jet= ty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) = at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.= java:609) at java.lang.Thread.run(Thread.java:745) 36) Thread[id= =3D1518, name=3DTEST-BasicDistributedZk2Test.test-seed#[6E953035E71A70BF]-E= ventThread, state=3DWAITING, group=3DTGRP-BasicDistributedZk2Test] = at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.lock= s.LockSupport.park(LockSupport.java:175) at java.util.concurrent.lo= cks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchron= izer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(Li= nkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$Even= tThread.run(ClientCnxn.java:494) 37) Thread[id=3D1561, name=3Dqtp39566789= 8-1561, state=3DRUNNABLE, group=3DTGRP-BasicDistributedZk2Test] at = sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.D= evPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch= .DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.n= io.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio= .ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.Selecto= rImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.Managed= Selector$SelectorProducer.select(ManagedSelector.java:233) at org.e= clipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.ja= va:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceCon= sume.produceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.j= etty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsu= me.java:102) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSel= ector.java:137) at org.eclipse.jetty.util.thread.QueuedThreadPool.r= unJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thread.Q= ueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Threa= d.run(Thread.java:745) 38) Thread[id=3D1501, name=3DTEST-BasicDistributed= Zk2Test.test-seed#[6E953035E71A70BF]-SendThread(127.0.0.1:39555), state=3DR= UNNABLE, group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.DevPol= lArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrappe= r.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorI= mpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImp= l.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.= select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSock= etNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zooke= eper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 39) Thread[id=3D1551= , name=3Dqtp1050973900-1551, state=3DTIMED_WAITING, group=3DTGRP-BasicDistr= ibutedZk2Test] at sun.misc.Unsafe.park(Native Method) at ja= va.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) = at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.= awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jet= ty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org= .eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.ja= va:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$80= 0(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.Queued= ThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run= (Thread.java:745) 40) Thread[id=3D1535, name=3DScheduler-764694638, state= =3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc= .Unsafe.park(Native Method) at java.util.concurrent.locks.LockSuppo= rt.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.Ab= stractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroni= zer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$= DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at jav= a.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Schedul= edThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolEx= ecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurren= t.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at jav= a.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617= ) at java.lang.Thread.run(Thread.java:745) 41) Thread[id=3D1512, = name=3DsearcherExecutor-628-thread-1, state=3DWAITING, group=3DTGRP-BasicDi= stributedZk2Test] at sun.misc.Unsafe.park(Native Method) at= java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) = at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.aw= ait(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.L= inkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.= concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) = at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja= va:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) = 42) Thread[id=3D1549, name=3Dqtp1050973900-1549, state=3DRUNNABLE, group= =3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.= poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPoll= ArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(D= evPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSel= ect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(Selecto= rImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:= 101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.selec= t(ManagedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector= $SelectorProducer.produce(ManagedSelector.java:181) at org.eclipse.= jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProdu= ceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.Execu= teProduceConsume.execute(ExecuteProduceConsume.java:102) at org.ecl= ipse.jetty.io.ManagedSelector.run(ManagedSelector.java:137) at org.= eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654= ) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThr= eadPool.java:572) at java.lang.Thread.run(Thread.java:745) 43) Th= read[id=3D1542, name=3Dqtp567249146-1542, state=3DTIMED_WAITING, group=3DTG= RP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) = at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Con= ditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at or= g.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queued= ThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadP= ool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.= thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.la= ng.Thread.run(Thread.java:745) 44) Thread[id=3D1550, name=3Dqtp1050973900= -1550-acceptor-0@6e1af8b4-ServerConnector@5cb534ad{HTTP/1.1,[http/1.1]}{127= .0.0.1:62809}, state=3DRUNNABLE, group=3DTGRP-BasicDistributedZk2Test] = at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at= sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422= ) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelI= mpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(Se= rverConnector.java:382) at org.eclipse.jetty.server.AbstractConnect= or$Acceptor.run(AbstractConnector.java:593) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org= .eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572= ) at java.lang.Thread.run(Thread.java:745) 45) Thread[id=3D1529, = name=3Dorg.eclipse.jetty.server.session.HashSessionManager@26b09f6Timer, st= ate=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.m= isc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSu= pport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks= .AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchr= onizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecut= or$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at = java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sche= duledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoo= lExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concur= rent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at = java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 617) at java.lang.Thread.run(Thread.java:745) 46) Thread[id=3D153= 8, name=3Dqtp567249146-1538-acceptor-0@2e5c45c8-ServerConnector@7e3b4032{HT= TP/1.1,[http/1.1]}{127.0.0.1:37004}, state=3DRUNNABLE, group=3DTGRP-BasicDi= stributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.accept0(Nat= ive Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSock= etChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accep= t(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.Ser= verConnector.accept(ServerConnector.java:382) at org.eclipse.jetty.= server.AbstractConnector$Acceptor.run(AbstractConnector.java:593) a= t org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.ja= va:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(Que= uedThreadPool.java:572) at java.lang.Thread.run(Thread.java:745) = 47) Thread[id=3D1517, name=3DTEST-BasicDistributedZk2Test.test-seed#[6E9530= 35E71A70BF]-SendThread(127.0.0.1:39555), state=3DRUNNABLE, group=3DTGRP-Bas= icDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Nativ= e Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrappe= r.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelec= torImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:= 97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientC= nxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThrea= d.run(ClientCnxn.java:1081) 48) Thread[id=3D1548, name=3DConnection evict= or, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at= java.lang.Thread.sleep(Native Method) at org.apache.http.impl.clie= nt.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at ja= va.lang.Thread.run(Thread.java:745) 49) Thread[id=3D1471, name=3DThread-3= 76, state=3DWAITING, group=3DTGRP-BasicDistributedZk2Test] at java.= lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.jav= a:1249) at java.lang.Thread.join(Thread.java:1323) at org.a= pache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:= 302) at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConf= ig(ZkTestServer.java:310) at org.apache.solr.cloud.ZkTestServer$2.r= un(ZkTestServer.java:491) 50) Thread[id=3D1554, name=3Dqtp1050973900-1554= , state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at s= un.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.Lo= ckSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.l= ocks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSy= nchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.= poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.= QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.ecli= pse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)= at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThre= adPool.java:609) at java.lang.Thread.run(Thread.java:745) 51) Thr= ead[id=3D1488, name=3Dqtp198503387-1488, state=3DTIMED_WAITING, group=3DTGR= P-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) = at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java= :215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Cond= itionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org= .eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedT= hreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPo= ol.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util.t= hread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.lan= g.Thread.run(Thread.java:745) 52) Thread[id=3D1560, name=3DConnection evi= ctor, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] = at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.cl= ient.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at = java.lang.Thread.run(Thread.java:745) 53) Thread[id=3D1537, name=3Dqtp567= 249146-1537, state=3DRUNNABLE, group=3DTGRP-BasicDistributedZk2Test] = at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio= .ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.n= io.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at = sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at su= n.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.Se= lectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.Ma= nagedSelector$SelectorProducer.select(ManagedSelector.java:233) at = org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelect= or.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProdu= ceConsume.produceAndRun(ExecuteProduceConsume.java:171) at org.ecli= pse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduce= Consume.java:102) at org.eclipse.jetty.io.ManagedSelector.run(Manag= edSelector.java:137) at org.eclipse.jetty.util.thread.QueuedThreadP= ool.runJob(QueuedThreadPool.java:654) at org.eclipse.jetty.util.thr= ead.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.= Thread.run(Thread.java:745) 54) Thread[id=3D1541, name=3Dqtp567249146-154= 1, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at = sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.L= ockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.= locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedS= ynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue= .poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread= .QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47= ) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThr= eadPool.java:609) at java.lang.Thread.run(Thread.java:745) 55) Th= read[id=3D1531, name=3Dqtp1344769124-1531, state=3DRUNNABLE, group=3DTGRP-B= asicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Nat= ive Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrap= per.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSel= ectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Selec= torImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) = at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(Managed= Selector.java:233) at org.eclipse.jetty.io.ManagedSelector$Selector= Producer.produce(ManagedSelector.java:181) at org.eclipse.jetty.uti= l.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume= .java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduce= Consume.run(ExecuteProduceConsume.java:156) at org.eclipse.jetty.ut= il.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654) at org= .eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572= ) at java.lang.Thread.run(Thread.java:745) 56) Thread[id=3D1557, = name=3Dorg.eclipse.jetty.server.session.HashSessionManager@57eb84b2Timer, s= tate=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.= misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockS= upport.parkNanos(LockSupport.java:215) at java.util.concurrent.lock= s.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynch= ronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecu= tor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at= java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sch= eduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPo= olExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concu= rrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at= java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java= :617) at java.lang.Thread.run(Thread.java:745) 57) Thread[id=3D15= 39, name=3Dqtp567249146-1539, state=3DTIMED_WAITING, group=3DTGRP-BasicDist= ributedZk2Test] at sun.misc.Unsafe.park(Native Method) at j= ava.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) = at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject= .awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.je= tty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at or= g.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.j= ava:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$8= 00(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.Queue= dThreadPool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.ru= n(Thread.java:745) 58) Thread[id=3D1496, name=3DConnection evictor, state= =3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at java.lan= g.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleCo= nnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.T= hread.run(Thread.java:745) 59) Thread[id=3D1524, name=3Dqtp1344769124-152= 4, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] at = sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.L= ockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.= locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedS= ynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue= .poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread= .QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at org.ecl= ipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47= ) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThr= eadPool.java:609) at java.lang.Thread.run(Thread.java:745) 60) Th= read[id=3D1525, name=3Dqtp1344769124-1525, state=3DTIMED_WAITING, group=3DT= GRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method)= at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.ja= va:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$Co= nditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at o= rg.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThread= Pool.access$800(QueuedThreadPool.java:47) at org.eclipse.jetty.util= .thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at java.l= ang.Thread.run(Thread.java:745) 61) Thread[id=3D1502, name=3DTEST-BasicDi= stributedZk2Test.test-seed#[6E953035E71A70BF]-EventThread, state=3DWAITING,= group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Nati= ve Method) at java.util.concurrent.locks.LockSupport.park(LockSuppo= rt.java:175) at java.util.concurrent.locks.AbstractQueuedSynchroniz= er$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at j= ava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) = at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:= 494) 62) Thread[id=3D1530, name=3Dqtp1344769124-1530, state=3DTIMED_WAITI= NG, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(N= ative Method) at java.util.concurrent.locks.LockSupport.parkNanos(L= ockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSy= nchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)= at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQue= ue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idle= JobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread= .QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org.eclip= se.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) = at java.lang.Thread.run(Thread.java:745) 63) Thread[id=3D1553, name= =3Dqtp1050973900-1553, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributed= Zk2Test] at sun.misc.Unsafe.park(Native Method) at java.uti= l.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at j= ava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN= anos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.uti= l.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclip= se.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546= ) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queu= edThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThread= Pool$3.run(QueuedThreadPool.java:609) at java.lang.Thread.run(Threa= d.java:745) 64) Thread[id=3D1484, name=3Dqtp198503387-1484, state=3DTIMED= _WAITING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.= park(Native Method) at java.util.concurrent.locks.LockSupport.parkN= anos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQu= euedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java= :2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingAr= rayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPoo= l.idleJobPoll(QueuedThreadPool.java:546) at org.eclipse.jetty.util.= thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47) at org= .eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609= ) at java.lang.Thread.run(Thread.java:745) 65) Thread[id=3D1508, = name=3DThread-387, state=3DWAITING, group=3DTGRP-BasicDistributedZk2Test] = at java.lang.Object.wait(Native Method) at java.lang.Object.= wait(Object.java:502) at org.apache.solr.core.CloserThread.run(Core= Container.java:1292) 66) Thread[id=3D1505, name=3DOverseerCollectionConfi= gSetProcessor-96516263976960004-127.0.0.1:43794_-n_0000000000, state=3DTIME= D_WAITING, group=3DOverseer collection creation process.] at sun.mi= sc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSup= port.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.= AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchro= nizer.java:2078) at org.apache.solr.cloud.DistributedQueue.peekElem= ents(DistributedQueue.java:343) at org.apache.solr.cloud.OverseerTa= skQueue.peekTopN(OverseerTaskQueue.java:235) at org.apache.solr.clo= ud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:225) at jav= a.lang.Thread.run(Thread.java:745) 67) Thread[id=3D1540, name=3Dqtp567249= 146-1540, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZk2Test] = at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.= locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.conc= urrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abstract= QueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArr= ayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util= .thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546) at = org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.= java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(Qu= euedThreadPool.java:609) at java.lang.Thread.run(Thread.java:745) = 68) Thread[id=3D1555, name=3Dqtp1050973900-1555, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native = Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSup= port.java:215) at java.util.concurrent.locks.AbstractQueuedSynchron= izer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) = at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.jav= a:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPol= l(QueuedThreadPool.java:546) at org.eclipse.jetty.util.thread.Queue= dThreadPool.access$800(QueuedThreadPool.java:47) at org.eclipse.jet= ty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609) at= java.lang.Thread.run(Thread.java:745) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 68 threads leaked from = SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test:=20 1) Thread[id=3D1475, name=3DProcessThread(sid:0 cport:-1):, state=3DWAIT= ING, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175= ) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueu= e.java:442) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequest= Processor.java:120) 2) Thread[id=3D1545, name=3Dorg.eclipse.jetty.server.session.HashSession= Manager@25fd8ffdTimer, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributed= Zk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecut= or.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:617) at java.lang.Thread.run(Thread.java:745) 3) Thread[id=3D1494, name=3Dqtp198503387-1494, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 4) Thread[id=3D1567, name=3Dqtp395667898-1567, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 5) Thread[id=3D1473, name=3DSessionTracker, state=3DTIMED_WAITING, group= =3DTGRP-BasicDistributedZk2Test] at java.lang.Object.wait(Native Method) at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTracke= rImpl.java:146) 6) Thread[id=3D1483, name=3Dqtp198503387-1483, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 7) Thread[id=3D1516, name=3DConnection evictor, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConn= ectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 8) Thread[id=3D1472, name=3DNIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, stat= e=3DRUNNABLE, group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223= ) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java= :98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCn= xnFactory.java:178) at java.lang.Thread.run(Thread.java:745) 9) Thread[id=3D1526, name=3Dqtp1344769124-1526-acceptor-0@5df1cecf-Serve= rConnector@34661a88{HTTP/1.1,[http/1.1]}{127.0.0.1:39555}, state=3DRUNNABLE= , group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.= java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 10) Thread[id=3D1572, name=3DConnection evictor, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConn= ectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 11) Thread[id=3D1504, name=3DOverseerStateUpdate-96516263976960004-127.0.= 0.1:43794_-n_0000000000, state=3DTIMED_WAITING, group=3DOverseer state upda= ter.] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.jav= a:164) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.jav= a:138) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.= java:196) at java.lang.Thread.run(Thread.java:745) 12) Thread[id=3D1565, name=3Dqtp395667898-1565, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 13) Thread[id=3D1487, name=3Dqtp198503387-1487, state=3DRUNNABLE, group= =3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223= ) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java= :98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(Man= agedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(Ma= nagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.pro= duceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.exe= cute(ExecuteProduceConsume.java:102) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:13= 7) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 14) Thread[id=3D1527, name=3Dqtp1344769124-1527, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 15) Thread[id=3D1536, name=3DConnection evictor, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConn= ectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 16) Thread[id=3D1562, name=3Dqtp395667898-1562-acceptor-0@5bead1a8-Server= Connector@205306da{HTTP/1.1,[http/1.1]}{127.0.0.1:40345}, state=3DRUNNABLE,= group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.= java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 17) Thread[id=3D1497, name=3DConnection evictor, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConn= ectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 18) Thread[id=3D1552, name=3Dqtp1050973900-1552, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 19) Thread[id=3D1515, name=3DConnection evictor, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConn= ectionEvictor.java:66) at java.lang.Thread.run(Thread.java:745) 20) Thread[id=3D1532, name=3Dqtp1344769124-1532, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 21) Thread[id=3D1490, name=3Dorg.eclipse.jetty.server.session.HashSession= Manager@3b500f87Timer, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributed= Zk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecut= or.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:617) at java.lang.Thread.run(Thread.java:745) 22) Thread[id=3D1506, name=3DOverseerHdfsCoreFailoverThread-9651626397696= 0004-127.0.0.1:43794_-n_0000000000, state=3DTIMED_WAITING, group=3DOverseer= Hdfs SolrCore Failover Thread.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(Over= seerAutoReplicaFailoverThread.java:137) at java.lang.Thread.run(Thread.java:745) 23) Thread[id=3D1568, name=3Dqtp395667898-1568, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 24) Thread[id=3D1566, name=3Dqtp395667898-1566, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 25) Thread[id=3D1543, name=3Dqtp567249146-1543, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 26) Thread[id=3D1556, name=3Dqtp1050973900-1556, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 27) Thread[id=3D1486, name=3Dqtp198503387-1486, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 28) Thread[id=3D1485, name=3Dqtp198503387-1485-acceptor-0@69afa4fc-Server= Connector@2281f6fb{HTTP/1.1,[http/1.1]}{127.0.0.1:43794}, state=3DRUNNABLE,= group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.= java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 29) Thread[id=3D1522, name=3Dqtp1344769124-1522, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 30) Thread[id=3D1564, name=3Dqtp395667898-1564, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 31) Thread[id=3D1544, name=3Dqtp567249146-1544, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 32) Thread[id=3D1523, name=3Dqtp1344769124-1523, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 33) Thread[id=3D1569, name=3Dorg.eclipse.jetty.server.session.HashSession= Manager@4f550564Timer, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributed= Zk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecut= or.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:617) at java.lang.Thread.run(Thread.java:745) 34) Thread[id=3D1563, name=3Dqtp395667898-1563, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 35) Thread[id=3D1482, name=3Dqtp198503387-1482, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 36) Thread[id=3D1518, name=3DTEST-BasicDistributedZk2Test.test-seed#[6E95= 3035E71A70BF]-EventThread, state=3DWAITING, group=3DTGRP-BasicDistributedZk= 2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175= ) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueu= e.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:= 494) 37) Thread[id=3D1561, name=3Dqtp395667898-1561, state=3DRUNNABLE, group= =3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223= ) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java= :98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(Man= agedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(Ma= nagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.pro= duceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.exe= cute(ExecuteProduceConsume.java:102) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:13= 7) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 38) Thread[id=3D1501, name=3DTEST-BasicDistributedZk2Test.test-seed#[6E95= 3035E71A70BF]-SendThread(127.0.0.1:39555), state=3DRUNNABLE, group=3DTGRP-B= asicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223= ) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java= :98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnS= ocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1= 081) 39) Thread[id=3D1551, name=3Dqtp1050973900-1551, state=3DTIMED_WAITING, g= roup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 40) Thread[id=3D1535, name=3DScheduler-764694638, state=3DTIMED_WAITING, = group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecut= or.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:617) at java.lang.Thread.run(Thread.java:745) 41) Thread[id=3D1512, name=3DsearcherExecutor-628-thread-1, state=3DWAITI= NG, group=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175= ) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueu= e.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecut= or.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:617) at java.lang.Thread.run(Thread.java:745) 42) Thread[id=3D1549, name=3Dqtp1050973900-1549, state=3DRUNNABLE, group= =3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223= ) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java= :98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(Man= agedSelector.java:233) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(Ma= nagedSelector.java:181) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.pro= duceAndRun(ExecuteProduceConsume.java:171) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.exe= cute(ExecuteProduceConsume.java:102) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:13= 7) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 43) Thread[id=3D1542, name=3Dqtp567249146-1542, state=3DTIMED_WAITING, gr= oup=3DTGRP-BasicDistributedZk2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueu= e.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(Queue= dThreadPool.java:546) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(Queued= ThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:609) at java.lang.Thread.run(Thread.java:745) 44) Thread[id=3D1550, name=3Dqtp1050973900-1550-acceptor-0@6e1af8b4-Serve= rConnector@5cb534ad{HTTP/1.1,[http/1.1]}{127.0.0.1:62809}, state=3DRUNNABLE= , group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.= java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 45) Thread[id=3D1529, name=3Dorg.eclipse.jetty.server.session.HashSession= Manager@26b09f6Timer, state=3DTIMED_WAITING, group=3DTGRP-BasicDistributedZ= k2Test] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.jav= a:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueu= e.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecut= or.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExec= utor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:617) at java.lang.Thread.run(Thread.java:745) 46) Thread[id=3D1538, name=3Dqtp567249146-1538-acceptor-0@2e5c45c8-Server= Connector@7e3b4032{HTTP/1.1,[http/1.1]}{127.0.0.1:37004}, state=3DRUNNABLE,= group=3DTGRP-BasicDistributedZk2Test] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.= java:382) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abstract= Connector.java:593) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThre= adPool.java:654) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThrea= dPool.java:572) at java.lang.Thread.run(Thread.java:745) 47) Thread[id=3D1517, name=3DTEST-BasicDistributedZk2Test.test-seed#[6E95= 3035E71A70BF]-SendThread(127.0.0.1:39555), state=3DRUNNABLE, group=3DTGRP-B= asicDistributedZk2Test] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223= ) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java= :98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeep [...truncated too long message...] rt/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/= test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_2F53DD7343E8D57-= 001/tempDir-001/node3/./testcollection_shard2_replica1/data;done=3Dfalse>>] [junit4] 2> 230061 INFO (coreCloseExecutor-810-thread-2) [n:127.0.0.1= :32984_solr c:testcollection s:shard2 r:core_node3 x:testcollection_shard2_= replica1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#= isShutdown=3Dtrue [junit4] 2> 230061 INFO (coreCloseExecutor-810-thread-2) [n:127.0.0.1= :32984_solr c:testcollection s:shard2 r:core_node3 x:testcollection_shard2_= replica1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/j= enkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J0/t= emp/solr.common.cloud.TestCollectionStateWatchers_2F53DD7343E8D57-001/tempD= ir-001/node3/./testcollection_shard2_replica1/data [junit4] 2> 230063 INFO (zkCallback-550-thread-1-processing-n:127.0.0= .1:32984_solr) [n:127.0.0.1:32984_solr ] o.a.s.c.OverseerElectionContext= I am going to be the leader 127.0.0.1:32984_solr [junit4] 2> 230063 INFO (zkCallback-550-thread-1-processing-n:127.0.0= .1:32984_solr) [n:127.0.0.1:32984_solr ] o.a.s.c.c.SolrZkClient makePath= : /overseer_elect/leader [junit4] 2> 230064 INFO (zkCallback-550-thread-2-processing-n:127.0.0= .1:32984_solr) [n:127.0.0.1:32984_solr ] o.a.s.c.c.ZkStateReader A live = node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged pat= h:/live_nodes], has occurred - updating... (live nodes size: [3]) [junit4] 2> 230066 ERROR (jetty-launcher-509-thread-1-EventThread) [n:= 127.0.0.1:39550_solr ] o.a.z.ClientCnxn Error while calling watcher=20 [junit4] 2> java.util.concurrent.RejectedExecutionException: Task org.= apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$47/= 1404419481@7eb4637d rejected from org.apache.solr.common.util.ExecutorUtil$= MDCAwareThreadPoolExecutor@21d8c3ff[Terminated, pool size =3D 9, active thr= eads =3D 0, queued tasks =3D 0, completed tasks =3D 420] [junit4] 2> =09at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.= rejectedExecution(ThreadPoolExecutor.java:2047) [junit4] 2> =09at java.util.concurrent.ThreadPoolExecutor.reject(Threa= dPoolExecutor.java:823) [junit4] 2> =09at java.util.concurrent.ThreadPoolExecutor.execute(Thre= adPoolExecutor.java:1369) [junit4] 2> =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThr= eadPoolExecutor.execute(ExecutorUtil.java:214) [junit4] 2> =09at java.util.concurrent.AbstractExecutorService.submit(= AbstractExecutorService.java:112) [junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$3.process(= SolrZkClient.java:266) [junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.processE= vent(ClientCnxn.java:522) [junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(Clie= ntCnxn.java:498) [junit4] 2> 230069 INFO (jetty-launcher-509-thread-4) [ ] o.e.j.s.= h.ContextHandler Stopped o.e.j.s.ServletContextHandler@958e03d{/solr,null,U= NAVAILABLE} [junit4] 2> 230069 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ = ] o.a.z.s.NIOServerCnxn caught end of stream exception [junit4] 2> EndOfStreamException: Unable to read additional data from = client sessionid 0x156e51ebf310015, likely client has closed socket [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOSe= rverCnxn.java:228) [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxnFactory.run= (NIOServerCnxnFactory.java:208) [junit4] 2> =09at java.lang.Thread.run(Thread.java:745) [junit4] 2> 231569 WARN (zkCallback-550-thread-2-processing-n:127.0.0= .1:32984_solr) [n:127.0.0.1:32984_solr ] o.a.s.c.c.ZkStateReader ZooKeep= er watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode =3D Sessio= n expired for /live_nodes] [junit4] 2> 231573 WARN (zkCallback-550-thread-1-processing-n:127.0.0= .1:32984_solr) [n:127.0.0.1:32984_solr ] o.a.s.c.LeaderElector=20 [junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredExcepti= on: KeeperErrorCode =3D Session expired for /overseer_elect/leader [junit4] 2> =09at org.apache.zookeeper.KeeperException.create(KeeperEx= ception.java:127) [junit4] 2> =09at org.apache.zookeeper.KeeperException.create(KeeperEx= ception.java:51) [junit4] 2> =09at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java= :783) [junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$10.execute= (SolrZkClient.java:503) [junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retryOper= ation(ZkCmdExecutor.java:60) [junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePath(S= olrZkClient.java:500) [junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePath(S= olrZkClient.java:458) [junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePath(S= olrZkClient.java:445) [junit4] 2> =09at org.apache.solr.cloud.OverseerElectionContext.runLea= derProcess(ElectionContext.java:724) [junit4] 2> =09at org.apache.solr.cloud.LeaderElector.runIamLeaderProc= ess(LeaderElector.java:170) [junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLeader= (LeaderElector.java:135) [junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$200(Leade= rElector.java:56) [junit4] 2> =09at org.apache.solr.cloud.LeaderElector$ElectionWatcher.= process(LeaderElector.java:348) [junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$3.lambda$p= rocess$0(SolrZkClient.java:266) [junit4] 2> =09at java.util.concurrent.Executors$RunnableAdapter.call(= Executors.java:511) [junit4] 2> =09at java.util.concurrent.FutureTask.run(FutureTask.java:= 266) [junit4] 2> =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThr= eadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) [junit4] 2> =09at java.util.concurrent.ThreadPoolExecutor.runWorker(Th= readPoolExecutor.java:1142) [junit4] 2> =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(T= hreadPoolExecutor.java:617) [junit4] 2> =09at java.lang.Thread.run(Thread.java:745) [junit4] 2> 231574 INFO (jetty-launcher-509-thread-2) [ ] o.e.j.s.= h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ba015e{/solr,null,U= NAVAILABLE} [junit4] 2> 231578 INFO (SUITE-TestCollectionStateWatchers-seed#[2F53= DD7343E8D57]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54= 669 54669 [junit4] 2> 231593 INFO (Thread-291) [ ] o.a.s.c.ZkTestServer conn= ecting to 127.0.0.1:54669 54669 [junit4] 2> 231596 WARN (Thread-291) [ ] o.a.s.c.ZkTestServer Watc= h limit violations:=20 [junit4] 2> Maximum concurrent create/delete watches above limit: [junit4] 2>=20 [junit4] 2> =098=09/solr/aliases.json [junit4] 2> =098=09/solr/clusterprops.json [junit4] 2> =097=09/solr/security.json [junit4] 2> =097=09/solr/configs/config [junit4] 2> =093=09/solr/collections/stateformat1/state.json [junit4] 2>=20 [junit4] 2> Maximum concurrent data watches above limit: [junit4] 2>=20 [junit4] 2> =0912=09/solr/collections/waitforstate/state.json [junit4] 2> =0912=09/solr/collections/falsepredicate/state.json [junit4] 2> =098=09/solr/clusterstate.json [junit4] 2> =097=09/solr/collections/testcollection/state.json [junit4] 2> =093=09/solr/collections/stateformat1/state.json [junit4] 2> =093=09/solr/overseer_elect/election/96516362254483467-127= .0.0.1:43617_solr-n_0000000001 [junit4] 2> =092=09/solr/collections/currentstate/state.json [junit4] 2> =092=09/solr/collections/delayed/state.json [junit4] 2>=20 [junit4] 2> Maximum concurrent children watches above limit: [junit4] 2>=20 [junit4] 2> =0962=09/solr/overseer/queue [junit4] 2> =0937=09/solr/overseer/collection-queue-work [junit4] 2> =0914=09/solr/overseer/queue-work [junit4] 2> =098=09/solr/collections [junit4] 2> =095=09/solr/live_nodes [junit4] 2>=20 [junit4] 2> 231596 INFO (SUITE-TestCollectionStateWatchers-seed#[2F53= DD7343E8D57]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jen= kins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-solrj/test/J0/tem= p/solr.common.cloud.TestCollectionStateWatchers_2F53DD7343E8D57-001 [junit4] 2> Sep 01, 2016 9:41:09 AM com.carrotsearch.randomizedtesting= .ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thre= ad(s). [junit4] 2> NOTE: test params are: codec=3DAsserting(Lucene62): {}, do= cValues:{}, maxPointsInLeafNode=3D1049, maxMBSortInHeap=3D6.500826147395175= , sim=3DRandomSimilarity(queryNorm=3Dtrue): {}, locale=3Dar-BH, timezone=3D= America/Boa_Vista [junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_102 (64-bi= t)/cpus=3D3,threads=3D1,free=3D203728808,total=3D482344960 [junit4] 2> NOTE: All tests run in this JVM: [SolrExampleEmbeddedTest,= SolrExampleStreamingTest, StreamingTest, TestToleratedUpdateError, FieldAn= alysisResponseTest, TestEmbeddedSolrServer, SolrDocumentTest, JdbcDriverTes= t, CollectionAdminRequestRequiredParamsTest, NoOpResponseParserTest, TestJa= vaBinCodec, SolrExampleXMLTest, TestHash, CloudSolrClientBuilderTest, TestS= pellCheckResponse, TestDocumentObjectBinder, TestSolrJErrorHandling, TestCo= reAdmin, StreamExpressionToExpessionTest, JettyWebappTest, TestDelegationTo= kenRequest, SolrExampleBinaryTest, TestBatchUpdate, TestSolrProperties, Sol= rParamTest, TermsResponseTest, MergeIndexesEmbeddedTest, TestNamedListCodec= , SolrQueryTest, ModifiableSolrParamsTest, TestXMLEscaping, DocumentAnalysi= sResponseTest, SolrExceptionTest, TestUpdateRequestCodec, ClientUtilsTest, = NamedListTest, AnlysisResponseBaseTest, GetByIdTest, SolrSchemalessExampleT= est, BasicHttpSolrClientTest, CloudSolrClientTest, LBHttpSolrClientBuilderT= est, TestCloudSolrClientConnections, GraphTest, JDBCStreamTest, SolrPingTes= t, TestCollectionAdminRequest, TestDelegationTokenResponse, TestSuggesterRe= sponse, TestCollectionStateWatchers] [junit4] Completed [85/85 (1!)] on J0 in 57.46s, 9 tests, 1 error <<< FA= ILURES! [...truncated 59465 lines...] ------=_Part_102_1908515820.1472723972266 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org ------=_Part_102_1908515820.1472723972266--