From builds-return-1359-archive-asf-public=cust-asf.ponee.io@lucene.apache.org Sat Dec 7 10:45:58 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id A831518065B for ; Sat, 7 Dec 2019 11:45:57 +0100 (CET) Received: (qmail 34061 invoked by uid 500); 7 Dec 2019 10:45:57 -0000 Mailing-List: contact builds-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list builds@lucene.apache.org Received: (qmail 34050 invoked by uid 99); 7 Dec 2019 10:45:56 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 Dec 2019 10:45:56 +0000 Received: from jenkins02.apache.org (jenkins02.apache.org [195.201.213.130]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTP id 6EE811004 for ; Sat, 7 Dec 2019 10:45:55 +0000 (UTC) Received: from jenkins02.apache.org (localhost.localdomain [127.0.0.1]) by jenkins02.apache.org (ASF Mail Server at jenkins02.apache.org) with ESMTP id 9583533E007C for ; Sat, 7 Dec 2019 10:45:54 +0000 (UTC) Date: Sat, 7 Dec 2019 10:42:12 +0000 (UTC) From: Apache Jenkins Server To: builds@lucene.apache.org Message-ID: <1624132049.16798.1575715554616.JavaMail.jenkins@jenkins02> In-Reply-To: <1838489379.16227.1575582596288.JavaMail.jenkins@jenkins02> References: <1838489379.16227.1575582596288.JavaMail.jenkins@jenkins02> Subject: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 2038 - Still Unstable MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_16797_2061718560.1575715553818" X-Jenkins-Job: Lucene-Solr-NightlyTests-master X-Jenkins-Result: UNSTABLE ------=_Part_16797_2061718560.1575715553818 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/2038/ 9 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonk= eySafeLeaderTest Error Message: access denied ("java.io.FilePermission" "<>" "execute") Stack Trace: java.security.AccessControlException: access denied ("java.io.FilePermissio= n" "<>" "execute") =09at __randomizedtesting.SeedInfo.seed([BA0A13FA8B8E0CDD]:0) =09at java.base/java.security.AccessControlContext.checkPermission(AccessCo= ntrolContext.java:472) =09at java.base/java.security.AccessController.checkPermission(AccessContro= ller.java:897) =09at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.j= ava:322) =09at java.base/java.lang.SecurityManager.checkExec(SecurityManager.java:57= 2) =09at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1096) =09at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) =09at org.apache.hadoop.util.Shell.runCommand(Shell.java:938) =09at org.apache.hadoop.util.Shell.run(Shell.java:901) =09at org.apache.hadoop.fs.DF.getFilesystem(DF.java:74) =09at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$Checke= dVolume.(NameNodeResourceChecker.java:70) =09at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.addDir= ToCheck(NameNodeResourceChecker.java:166) =09at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.= (NameNodeResourceChecker.java:135) =09at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServic= es(FSNamesystem.java:1168) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(N= ameNode.java:791) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.j= ava:717) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:= 940) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:= 913) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNo= de.java:1646) =09at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.j= ava:1314) =09at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSClu= ster.java:1083) =09at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniD= FSCluster.java:958) =09at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSClust= er.java:890) =09at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:518) =09at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.ja= va:477) =09at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:= 134) =09at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:= 66) =09at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.setupClass(H= dfsChaosMonkeySafeLeaderTest.java:42) =09at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nativ= e Method) =09at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Native= MethodAccessorImpl.java:62) =09at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(De= legatingMethodAccessorImpl.java:43) =09at java.base/java.lang.reflect.Method.invoke(Method.java:566) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1750) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random= izedRunner.java:878) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:894) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:368) =09at java.base/java.lang.Thread.run(Thread.java:834) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonk= eySafeLeaderTest Error Message: 15 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosM= onkeySafeLeaderTest: 1) Thread[id=3D9605, name=3Dqtp593746108-9605-acce= ptor-0@244efd87-ServerConnector@471c51e4{HTTP/1.1,[http/1.1]}{localhost.loc= aldomain:43089}, state=3DRUNNABLE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTe= st] at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept0(= Native Method) at java.base@11.0.4/sun.nio.ch.ServerSocketChannelIm= pl.accept(ServerSocketChannelImpl.java:533) at java.base@11.0.4/sun= .nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) = at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnect= or.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acc= eptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.uti= l.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) at app/= /org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool= .java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834= ) 2) Thread[id=3D9601, name=3Dpool-8-thread-1, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.i= nternal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.ut= il.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at = java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$Cond= itionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at jav= a.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQ= ueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.= 4/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sc= heduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.c= oncurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) = at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(Threa= dPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.T= hreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.b= ase@11.0.4/java.lang.Thread.run(Thread.java:834) 3) Thread[id=3D9602, na= me=3Dqtp593746108-9602, state=3DRUNNABLE, group=3DTGRP-HdfsChaosMonkeySafeL= eaderTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method)= at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSel= ectorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.loc= kAndDoSelect(SelectorImpl.java:124) at java.base@11.0.4/sun.nio.ch.= SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jett= y.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464) = at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(M= anagedSelector.java:401) at app//org.eclipse.jetty.util.thread.stra= tegy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//or= g.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKil= l.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatY= ouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jett= y.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) = at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$160/0x000000010037= fc40.run(Unknown Source) at app//org.eclipse.jetty.util.thread.Queu= edThreadPool.runJob(QueuedThreadPool.java:781) at app//org.eclipse.= jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) = at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 4) Threa= d[id=3D9604, name=3Dqtp593746108-9604, state=3DTIMED_WAITING, group=3DTGRP-= HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.mis= c.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurre= nt.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@1= 1.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject= .awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclip= se.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(Q= ueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.Que= uedThreadPool$Runner.run(QueuedThreadPool.java:889) at java.base@11= .0.4/java.lang.Thread.run(Thread.java:834) 5) Thread[id=3D9608, name=3Dq= tp593746108-9608, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLe= aderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native = Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.= parkNanos(LockSupport.java:234) at java.base@11.0.4/java.util.concu= rrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQ= ueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.Blockin= gArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.j= etty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:= 850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.= run(QueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread= .run(Thread.java:834) 6) Thread[id=3D9609, name=3Dqtp593746108-9609, sta= te=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at = java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at ja= va.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport= .java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQ= ueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.jav= a:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(Bloc= kingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.Queu= edThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850) at app//= org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.= java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)= 7) Thread[id=3D9607, name=3Dqtp593746108-9607, state=3DTIMED_WAITING, g= roup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.= internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.u= til.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at= java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$Con= ditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ap= p//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:3= 92) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.i= dleJobPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.uti= l.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889) at = java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 8) Thread[id=3D95= 99, name=3DTimer for 'NameNode' metrics system, state=3DTIMED_WAITING, grou= p=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/java.la= ng.Object.wait(Native Method) at java.base@11.0.4/java.util.TimerTh= read.mainLoop(Timer.java:553) at java.base@11.0.4/java.util.TimerTh= read.run(Timer.java:506) 9) Thread[id=3D9616, name=3DFSEditLogAsync, sta= te=3DWAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.b= ase@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.bas= e@11.0.4/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) = at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) = at java.base@11.0.4/java.util.concurrent.ArrayBlockingQueue.take(ArrayBlock= ingQueue.java:417) at app//org.apache.hadoop.hdfs.server.namenode.F= SEditLogAsync.dequeueEdit(FSEditLogAsync.java:221) at app//org.apac= he.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSEditLogAsync.java:229) = at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 10) Thr= ead[id=3D9618, name=3DSocket Reader #1 for port 43727, state=3DRUNNABLE, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/sun.n= io.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPol= lSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11.= 0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) = at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) = at app//org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Serve= r.java:1097) at app//org.apache.hadoop.ipc.Server$Listener$Reader.r= un(Server.java:1076) 11) Thread[id=3D9603, name=3Dqtp593746108-9603, stat= e=3DRUNNABLE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.b= ase@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4= /sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) = at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.ja= va:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(Selector= Impl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$Selecto= rProducer.select(ManagedSelector.java:464) at app//org.eclipse.jett= y.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401) = at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceT= ask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.= strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//= org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYou= Kill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWh= atYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jett= y.io.ManagedSelector$$Lambda$160/0x000000010037fc40.run(Unknown Source) = at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedTh= readPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThre= adPool$Runner.run(QueuedThreadPool.java:917) at java.base@11.0.4/ja= va.lang.Thread.run(Thread.java:834) 12) Thread[id=3D9610, name=3DSession-= HouseKeeper-47c0f955, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySa= feLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Nat= ive Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupp= ort.parkNanos(LockSupport.java:234) at java.base@11.0.4/java.util.c= oncurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abstr= actQueuedSynchronizer.java:2123) at java.base@11.0.4/java.util.conc= urrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoo= lExecutor.java:1182) at java.base@11.0.4/java.util.concurrent.Sched= uledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.ja= va:899) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor= .getTask(ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.uti= l.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) = at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run= (ThreadPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.= run(Thread.java:834) 13) Thread[id=3D9606, name=3Dqtp593746108-9606, stat= e=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at j= ava.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at jav= a.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.= java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQu= euedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java= :2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(Block= ingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.Queue= dThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850) at app//o= rg.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.j= ava:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) = 14) Thread[id=3D9619, name=3DIPC Server idle connection scanner for port = 43727, state=3DWAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] = at java.base@11.0.4/java.lang.Object.wait(Native Method) at java.b= ase@11.0.4/java.lang.Object.wait(Object.java:328) at java.base@11.0= .4/java.util.TimerThread.mainLoop(Timer.java:527) at java.base@11.0= .4/java.util.TimerThread.run(Timer.java:506) 15) Thread[id=3D9600, name= =3Dorg.apache.hadoop.util.JvmPauseMonitor$Monitor@545a9cb8, state=3DTIMED_W= AITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11= .0.4/java.lang.Thread.sleep(Native Method) at app//org.apache.hadoo= p.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:192) at jav= a.base@11.0.4/java.lang.Thread.run(Thread.java:834) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 15 threads leaked from = SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:=20 1) Thread[id=3D9605, name=3Dqtp593746108-9605-acceptor-0@244efd87-Server= Connector@471c51e4{HTTP/1.1,[http/1.1]}{localhost.localdomain:43089}, state= =3DRUNNABLE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept0(Nati= ve Method) at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept(Serve= rSocketChannelImpl.java:533) at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept(Serve= rSocketChannelImpl.java:285) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConne= ctor.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abs= tractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 2) Thread[id=3D9601, name=3Dpool-8-thread-1, state=3DTIMED_WAITING, grou= p=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getTask= (ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWork= er(ThreadPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.= run(ThreadPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 3) Thread[id=3D9602, name=3Dqtp593746108-9602, state=3DRUNNABLE, group= =3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.selec= t(ManagedSelector.java:464) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produ= ce(ManagedSelector.java:401) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doPro= duce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryPr= oduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$160/0x00000001= 0037fc40.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 4) Thread[id=3D9604, name=3Dqtp593746108-9604, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 5) Thread[id=3D9608, name=3Dqtp593746108-9608, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 6) Thread[id=3D9609, name=3Dqtp593746108-9609, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 7) Thread[id=3D9607, name=3Dqtp593746108-9607, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 8) Thread[id=3D9599, name=3DTimer for 'NameNode' metrics system, state= =3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/java.lang.Object.wait(Native Method) at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:553) at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) 9) Thread[id=3D9616, name=3DFSEditLogAsync, state=3DWAITING, group=3DTGR= P-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.park(Loc= kSupport.java:194) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11.0.4/java.util.concurrent.ArrayBlockingQueue.take(Ar= rayBlockingQueue.java:417) at app//org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.deque= ueEdit(FSEditLogAsync.java:221) at app//org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(F= SEditLogAsync.java:229) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 10) Thread[id=3D9618, name=3DSocket Reader #1 for port 43727, state=3DRUN= NABLE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Serv= er.java:1097) at app//org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.jav= a:1076) 11) Thread[id=3D9603, name=3Dqtp593746108-9603, state=3DRUNNABLE, group= =3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.selec= t(ManagedSelector.java:464) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produ= ce(ManagedSelector.java:401) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doPro= duce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryPr= oduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$160/0x00000001= 0037fc40.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 12) Thread[id=3D9610, name=3DSession-HouseKeeper-47c0f955, state=3DTIMED_= WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getTask= (ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWork= er(ThreadPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.= run(ThreadPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 13) Thread[id=3D9606, name=3Dqtp593746108-9606, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 14) Thread[id=3D9619, name=3DIPC Server idle connection scanner for port = 43727, state=3DWAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/java.lang.Object.wait(Native Method) at java.base@11.0.4/java.lang.Object.wait(Object.java:328) at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:527) at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) 15) Thread[id=3D9600, name=3Dorg.apache.hadoop.util.JvmPauseMonitor$Monit= or@545a9cb8, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderT= est] at java.base@11.0.4/java.lang.Thread.sleep(Native Method) at app//org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPause= Monitor.java:192) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) =09at __randomizedtesting.SeedInfo.seed([BA0A13FA8B8E0CDD]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonk= eySafeLeaderTest Error Message: There are still zombie threads that couldn't be terminated: 1) Thread[id= =3D9605, name=3Dqtp593746108-9605, state=3DTIMED_WAITING, group=3DTGRP-Hdfs= ChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Un= safe.park(Native Method) at java.base@11.0.4/java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11.0.= 4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awa= itNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.j= etty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at a= pp//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(Queue= dThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedT= hreadPool$Runner.run(QueuedThreadPool.java:889) at java.base@11.0.4= /java.lang.Thread.run(Thread.java:834) 2) Thread[id=3D9601, name=3Dpool-= 8-thread-1, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTe= st] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method= ) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNa= nos(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.= locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedS= ynchronizer.java:2123) at java.base@11.0.4/java.util.concurrent.Sch= eduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.= java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledThread= PoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) = at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getTask(T= hreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.concurre= nt.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at ja= va.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo= lExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(Thread= .java:834) 3) Thread[id=3D9602, name=3Dqtp593746108-9602, state=3DRUNNAB= LE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4= /sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.c= h.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.ba= se@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) = at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:= 141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.= select(ManagedSelector.java:464) at app//org.eclipse.jetty.io.Manag= edSelector$SelectorProducer.produce(ManagedSelector.java:401) at ap= p//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWha= tYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.E= atWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclips= e.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:= 168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.= produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.Manag= edSelector$$Lambda$160/0x000000010037fc40.run(Unknown Source) at ap= p//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.j= ava:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Run= ner.run(QueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Th= read.run(Thread.java:834) 4) Thread[id=3D9604, name=3Dqtp593746108-9604,= state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] = at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) a= t java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSup= port.java:234) at java.base@11.0.4/java.util.concurrent.locks.Abstr= actQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer= .java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(= BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.= QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850) at a= pp//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadP= ool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:= 834) 5) Thread[id=3D9608, name=3Dqtp593746108-9608, state=3DTIMED_WAITIN= G, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/= jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/ja= va.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) = at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer= $ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) a= t app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.ja= va:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runn= er.idleJobPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty= .util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889) = at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 6) Thread[id= =3D9609, name=3Dqtp593746108-9609, state=3DTIMED_WAITING, group=3DTGRP-Hdfs= ChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Un= safe.park(Native Method) at java.base@11.0.4/java.util.concurrent.l= ocks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11.0.= 4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awa= itNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.j= etty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at a= pp//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(Queue= dThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedT= hreadPool$Runner.run(QueuedThreadPool.java:889) at java.base@11.0.4= /java.lang.Thread.run(Thread.java:834) 7) Thread[id=3D9607, name=3Dqtp59= 3746108-9607, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeader= Test] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Meth= od) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.park= Nanos(LockSupport.java:234) at java.base@11.0.4/java.util.concurren= t.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueue= dSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArr= ayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty= .util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850)= at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(= QueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run= (Thread.java:834) 8) Thread[id=3D9599, name=3DTimer for 'NameNode' metri= cs system, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTes= t] at java.base@11.0.4/java.lang.Object.wait(Native Method) = at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:553) = at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) 9) Thread= [id=3D9618, name=3DSocket Reader #1 for port 43727, state=3DRUNNABLE, group= =3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/java.lan= g.Thread.holdsLock(Native Method) at java.base@11.0.4/sun.nio.ch.Se= lectorImpl.processDeregisterQueue(SelectorImpl.java:245) at java.ba= se@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:136)= at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Select= orImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select= (SelectorImpl.java:141) at app//org.apache.hadoop.ipc.Server$Listen= er$Reader.doRunLoop(Server.java:1097) at app//org.apache.hadoop.ipc= .Server$Listener$Reader.run(Server.java:1076) 10) Thread[id=3D9603, name= =3Dqtp593746108-9603, state=3DRUNNABLE, group=3DTGRP-HdfsChaosMonkeySafeLea= derTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) = at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelec= torImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockA= ndDoSelect(SelectorImpl.java:124) at java.base@11.0.4/sun.nio.ch.Se= lectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.= io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464) = at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(Man= agedSelector.java:401) at app//org.eclipse.jetty.util.thread.strate= gy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.= eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.= java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYou= Kill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.= util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) = at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$160/0x000000010037fc= 40.run(Unknown Source) at app//org.eclipse.jetty.util.thread.Queued= ThreadPool.runJob(QueuedThreadPool.java:781) at app//org.eclipse.je= tty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) = at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 11) Thread[= id=3D9610, name=3DSession-HouseKeeper-47c0f955, state=3DTIMED_WAITING, grou= p=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.int= ernal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util= .concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at ja= va.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$Condit= ionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.= base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQue= ue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.4/= java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sche= duledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.con= current.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at= java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadP= oolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.Thr= eadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.bas= e@11.0.4/java.lang.Thread.run(Thread.java:834) 12) Thread[id=3D9606, name= =3Dqtp593746108-9606, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySa= feLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Nat= ive Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupp= ort.parkNanos(LockSupport.java:234) at java.base@11.0.4/java.util.c= oncurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abstr= actQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.Blo= ckingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclip= se.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.j= ava:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Run= ner.run(QueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Th= read.run(Thread.java:834) 13) Thread[id=3D9619, name=3DIPC Server idle co= nnection scanner for port 43727, state=3DWAITING, group=3DTGRP-HdfsChaosMon= keySafeLeaderTest] at java.base@11.0.4/java.lang.Object.wait(Native= Method) at java.base@11.0.4/java.lang.Object.wait(Object.java:328)= at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:527)= at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie = threads that couldn't be terminated: 1) Thread[id=3D9605, name=3Dqtp593746108-9605, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 2) Thread[id=3D9601, name=3Dpool-8-thread-1, state=3DTIMED_WAITING, grou= p=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getTask= (ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWork= er(ThreadPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.= run(ThreadPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 3) Thread[id=3D9602, name=3Dqtp593746108-9602, state=3DRUNNABLE, group= =3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.selec= t(ManagedSelector.java:464) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produ= ce(ManagedSelector.java:401) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doPro= duce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryPr= oduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$160/0x00000001= 0037fc40.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 4) Thread[id=3D9604, name=3Dqtp593746108-9604, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 5) Thread[id=3D9608, name=3Dqtp593746108-9608, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 6) Thread[id=3D9609, name=3Dqtp593746108-9609, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 7) Thread[id=3D9607, name=3Dqtp593746108-9607, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 8) Thread[id=3D9599, name=3DTimer for 'NameNode' metrics system, state= =3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/java.lang.Object.wait(Native Method) at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:553) at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) 9) Thread[id=3D9618, name=3DSocket Reader #1 for port 43727, state=3DRUN= NABLE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/java.lang.Thread.holdsLock(Native Method) at java.base@11.0.4/sun.nio.ch.SelectorImpl.processDeregisterQueue(= SelectorImpl.java:245) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:136) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Serv= er.java:1097) at app//org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.jav= a:1076) 10) Thread[id=3D9603, name=3Dqtp593746108-9603, state=3DRUNNABLE, group= =3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.selec= t(ManagedSelector.java:464) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produ= ce(ManagedSelector.java:401) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doPro= duce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryPr= oduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$160/0x00000001= 0037fc40.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 11) Thread[id=3D9610, name=3DSession-HouseKeeper-47c0f955, state=3DTIMED_= WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getTask= (ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWork= er(ThreadPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.= run(ThreadPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 12) Thread[id=3D9606, name=3Dqtp593746108-9606, state=3DTIMED_WAITING, gr= oup=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 13) Thread[id=3D9619, name=3DIPC Server idle connection scanner for port = 43727, state=3DWAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] at java.base@11.0.4/java.lang.Object.wait(Native Method) at java.base@11.0.4/java.lang.Object.wait(Object.java:328) at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:527) at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) =09at __randomizedtesting.SeedInfo.seed([BA0A13FA8B8E0CDD]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsBasicDist= ributedZkTest Error Message: access denied ("java.io.FilePermission" "<>" "execute") Stack Trace: java.security.AccessControlException: access denied ("java.io.FilePermissio= n" "<>" "execute") =09at __randomizedtesting.SeedInfo.seed([BA0A13FA8B8E0CDD]:0) =09at java.base/java.security.AccessControlContext.checkPermission(AccessCo= ntrolContext.java:472) =09at java.base/java.security.AccessController.checkPermission(AccessContro= ller.java:897) =09at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.j= ava:322) =09at java.base/java.lang.SecurityManager.checkExec(SecurityManager.java:57= 2) =09at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1096) =09at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071) =09at org.apache.hadoop.util.Shell.runCommand(Shell.java:938) =09at org.apache.hadoop.util.Shell.run(Shell.java:901) =09at org.apache.hadoop.fs.DF.getFilesystem(DF.java:74) =09at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$Checke= dVolume.(NameNodeResourceChecker.java:70) =09at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.addDir= ToCheck(NameNodeResourceChecker.java:166) =09at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.= (NameNodeResourceChecker.java:135) =09at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServic= es(FSNamesystem.java:1168) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(N= ameNode.java:791) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.j= ava:717) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:= 940) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:= 913) =09at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNo= de.java:1646) =09at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.j= ava:1314) =09at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSClu= ster.java:1083) =09at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniD= FSCluster.java:958) =09at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSClust= er.java:890) =09at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:518) =09at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.ja= va:477) =09at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:= 134) =09at org.apache.solr.cloud.hdfs.HdfsTestUtil.setupClass(HdfsTestUtil.java:= 66) =09at org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest.setupClass(Hdfs= BasicDistributedZkTest.java:41) =09at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nativ= e Method) =09at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Native= MethodAccessorImpl.java:62) =09at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(De= legatingMethodAccessorImpl.java:43) =09at java.base/java.lang.reflect.Method.invoke(Method.java:566) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1750) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random= izedRunner.java:878) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:894) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:368) =09at java.base/java.lang.Thread.run(Thread.java:834) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsBasicDist= ributedZkTest Error Message: 15 threads leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsBasicD= istributedZkTest: 1) Thread[id=3D3512, name=3DSocket Reader #1 for port= 35815, state=3DRUNNABLE, group=3DTGRP-HdfsBasicDistributedZkTest] = at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.ba= se@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)= at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Select= orImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select= (SelectorImpl.java:141) at app//org.apache.hadoop.ipc.Server$Listen= er$Reader.doRunLoop(Server.java:1097) at app//org.apache.hadoop.ipc= .Server$Listener$Reader.run(Server.java:1076) 2) Thread[id=3D3499, name= =3Dqtp1258237229-3499, state=3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistrib= utedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Nativ= e Method) at java.base@11.0.4/java.util.concurrent.locks.LockSuppor= t.parkNanos(LockSupport.java:234) at java.base@11.0.4/java.util.con= current.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abstrac= tQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.Block= ingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse= .jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.jav= a:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runne= r.run(QueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thre= ad.run(Thread.java:834) 3) Thread[id=3D3496, name=3Dqtp1258237229-3496, = state=3DRUNNABLE, group=3DTGRP-HdfsBasicDistributedZkTest] at java.= base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.= 4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) = at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.j= ava:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(Selecto= rImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$Select= orProducer.select(ManagedSelector.java:464) at app//org.eclipse.jet= ty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:401) = at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce= Task(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread= .strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app/= /org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYo= uKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatW= hatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jet= ty.io.ManagedSelector$$Lambda$201/0x0000000100482440.run(Unknown Source) = at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedT= hreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThr= eadPool$Runner.run(QueuedThreadPool.java:917) at java.base@11.0.4/j= ava.lang.Thread.run(Thread.java:834) 4) Thread[id=3D3513, name=3DIPC Ser= ver idle connection scanner for port 35815, state=3DWAITING, group=3DTGRP-H= dfsBasicDistributedZkTest] at java.base@11.0.4/java.lang.Object.wai= t(Native Method) at java.base@11.0.4/java.lang.Object.wait(Object.j= ava:328) at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.j= ava:527) at java.base@11.0.4/java.util.TimerThread.run(Timer.java:5= 06) 5) Thread[id=3D3500, name=3Dqtp1258237229-3500, state=3DTIMED_WAITIN= G, group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk= .internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.= util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) a= t java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$Co= nditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at a= pp//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:= 392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.= idleJobPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.ut= il.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889) at= java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 6) Thread[id=3D3= 493, name=3DTimer for 'NameNode' metrics system, state=3DTIMED_WAITING, gro= up=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/java.lang= .Object.wait(Native Method) at java.base@11.0.4/java.util.TimerThre= ad.mainLoop(Timer.java:553) at java.base@11.0.4/java.util.TimerThre= ad.run(Timer.java:506) 7) Thread[id=3D3494, name=3Dorg.apache.hadoop.uti= l.JvmPauseMonitor$Monitor@20c40675, state=3DTIMED_WAITING, group=3DTGRP-Hdf= sBasicDistributedZkTest] at java.base@11.0.4/java.lang.Thread.sleep= (Native Method) at app//org.apache.hadoop.util.JvmPauseMonitor$Moni= tor.run(JvmPauseMonitor.java:192) at java.base@11.0.4/java.lang.Thr= ead.run(Thread.java:834) 8) Thread[id=3D3501, name=3Dqtp1258237229-3501,= state=3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistributedZkTest] at= java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at j= ava.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSuppor= t.java:234) at java.base@11.0.4/java.util.concurrent.locks.Abstract= QueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.ja= va:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(Blo= ckingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.Que= uedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:850) at app/= /org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool= .java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834= ) 9) Thread[id=3D3503, name=3Dqtp1258237229-3503, state=3DTIMED_WAITING,= group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.i= nternal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.ut= il.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at = java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$Cond= itionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app= //org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:39= 2) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.id= leJobPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util= .thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889) at j= ava.base@11.0.4/java.lang.Thread.run(Thread.java:834) 10) Thread[id=3D349= 7, name=3Dqtp1258237229-3497, state=3DRUNNABLE, group=3DTGRP-HdfsBasicDistr= ibutedZkTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Meth= od) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPoll= SelectorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.= lockAndDoSelect(SelectorImpl.java:124) at java.base@11.0.4/sun.nio.= ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.j= etty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:464) = at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produc= e(ManagedSelector.java:401) at app//org.eclipse.jetty.util.thread.s= trategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app/= /org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYou= Kill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWh= atYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.j= etty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) = at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$201/0x000000010= 0482440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.Q= ueuedThreadPool.runJob(QueuedThreadPool.java:781) at app//org.eclip= se.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)= at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 11) Th= read[id=3D3498, name=3Dqtp1258237229-3498-acceptor-0@5ebc708c-ServerConnect= or@5e86a353{HTTP/1.1,[http/1.1]}{localhost.localdomain:41835}, state=3DRUNN= ABLE, group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/= sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at java.b= ase@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImp= l.java:533) at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.= accept(ServerSocketChannelImpl.java:285) at app//org.eclipse.jetty.= server.ServerConnector.accept(ServerConnector.java:385) at app//org= .eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java= :648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob= (QueuedThreadPool.java:781) at app//org.eclipse.jetty.util.thread.Q= ueuedThreadPool$Runner.run(QueuedThreadPool.java:917) at java.base@= 11.0.4/java.lang.Thread.run(Thread.java:834) 12) Thread[id=3D3495, name= =3Dpool-3-thread-1, state=3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistribute= dZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native M= ethod) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.p= arkNanos(LockSupport.java:234) at java.base@11.0.4/java.util.concur= rent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQu= euedSynchronizer.java:2123) at java.base@11.0.4/java.util.concurren= t.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExec= utor.java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledT= hreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:89= 9) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getT= ask(ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.con= current.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) = at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(Thre= adPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(T= hread.java:834) 13) Thread[id=3D3502, name=3Dqtp1258237229-3502, state=3D= TIMED_WAITING, group=3DTGRP-HdfsBasicDistributedZkTest] at java.bas= e@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@= 11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:23= 4) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSyn= chronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) = at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.idleJobPoll(QueuedThreadPool.java:850) at app//org.ecli= pse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889= ) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 14) T= hread[id=3D3504, name=3DSession-HouseKeeper-1b37db29, state=3DTIMED_WAITING= , group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.= internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.u= til.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at= java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$Con= ditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at ja= va.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWork= Queue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0= .4/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(S= cheduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.= concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) = at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(Thre= adPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.= ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.= base@11.0.4/java.lang.Thread.run(Thread.java:834) 15) Thread[id=3D3510, n= ame=3DFSEditLogAsync, state=3DWAITING, group=3DTGRP-HdfsBasicDistributedZkT= est] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Metho= d) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.park(= LockSupport.java:194) at java.base@11.0.4/java.util.concurrent.lock= s.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchroniz= er.java:2081) at java.base@11.0.4/java.util.concurrent.ArrayBlockin= gQueue.take(ArrayBlockingQueue.java:417) at app//org.apache.hadoop.= hdfs.server.namenode.FSEditLogAsync.dequeueEdit(FSEditLogAsync.java:221) = at app//org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(FSE= ditLogAsync.java:229) at java.base@11.0.4/java.lang.Thread.run(Thre= ad.java:834) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 15 threads leaked from = SUITE scope at org.apache.solr.cloud.hdfs.HdfsBasicDistributedZkTest:=20 1) Thread[id=3D3512, name=3DSocket Reader #1 for port 35815, state=3DRUN= NABLE, group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Serv= er.java:1097) at app//org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.jav= a:1076) 2) Thread[id=3D3499, name=3Dqtp1258237229-3499, state=3DTIMED_WAITING, g= roup=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 3) Thread[id=3D3496, name=3Dqtp1258237229-3496, state=3DRUNNABLE, group= =3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.selec= t(ManagedSelector.java:464) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produ= ce(ManagedSelector.java:401) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doPro= duce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryPr= oduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$201/0x00000001= 00482440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 4) Thread[id=3D3513, name=3DIPC Server idle connection scanner for port = 35815, state=3DWAITING, group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/java.lang.Object.wait(Native Method) at java.base@11.0.4/java.lang.Object.wait(Object.java:328) at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:527) at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) 5) Thread[id=3D3500, name=3Dqtp1258237229-3500, state=3DTIMED_WAITING, g= roup=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 6) Thread[id=3D3493, name=3DTimer for 'NameNode' metrics system, state= =3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/java.lang.Object.wait(Native Method) at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:553) at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) 7) Thread[id=3D3494, name=3Dorg.apache.hadoop.util.JvmPauseMonitor$Monit= or@20c40675, state=3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistributedZkTest= ] at java.base@11.0.4/java.lang.Thread.sleep(Native Method) at app//org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPause= Monitor.java:192) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 8) Thread[id=3D3501, name=3Dqtp1258237229-3501, state=3DTIMED_WAITING, g= roup=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 9) Thread[id=3D3503, name=3Dqtp1258237229-3503, state=3DTIMED_WAITING, g= roup=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 10) Thread[id=3D3497, name=3Dqtp1258237229-3497, state=3DRUNNABLE, group= =3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSele= ctorImpl.java:120) at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAndDoSelect(Selecto= rImpl.java:124) at java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.jav= a:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.selec= t(ManagedSelector.java:464) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produ= ce(ManagedSelector.java:401) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doPro= duce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryPr= oduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produ= ce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$201/0x00000001= 00482440.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 11) Thread[id=3D3498, name=3Dqtp1258237229-3498-acceptor-0@5ebc708c-Serve= rConnector@5e86a353{HTTP/1.1,[http/1.1]}{localhost.localdomain:41835}, stat= e=3DRUNNABLE, group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept0(Nati= ve Method) at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept(Serve= rSocketChannelImpl.java:533) at java.base@11.0.4/sun.nio.ch.ServerSocketChannelImpl.accept(Serve= rSocketChannelImpl.java:285) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConne= ctor.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(Abs= tractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queue= dThreadPool.java:781) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 12) Thread[id=3D3495, name=3Dpool-3-thread-1, state=3DTIMED_WAITING, grou= p=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getTask= (ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWork= er(ThreadPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.= run(ThreadPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 13) Thread[id=3D3502, name=3Dqtp1258237229-3502, state=3DTIMED_WAITING, g= roup=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArra= yQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJ= obPoll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(Q= ueuedThreadPool.java:889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 14) Thread[id=3D3504, name=3DSession-HouseKeeper-1b37db29, state=3DTIMED_= WAITING, group=3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNano= s(LockSupport.java:234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.4/java.util.concurrent.ScheduledThreadPoolExecuto= r$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.getTask= (ThreadPoolExecutor.java:1054) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor.runWork= er(ThreadPoolExecutor.java:1114) at java.base@11.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.= run(ThreadPoolExecutor.java:628) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 15) Thread[id=3D3510, name=3DFSEditLogAsync, state=3DWAITING, group=3DTGR= P-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.concurrent.locks.LockSupport.park(Loc= kSupport.java:194) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11.0.4/java.util.concurrent.ArrayBlockingQueue.take(Ar= rayBlockingQueue.java:417) at app//org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.deque= ueEdit(FSEditLogAsync.java:221) at app//org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.run(F= SEditLogAsync.java:229) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) =09at __randomizedtesting.SeedInfo.seed([BA0A13FA8B8E0CDD]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsBasicDist= ributedZkTest Error Message: There are still zombie threads that couldn't be terminated: 1) Thread[id= =3D3512, name=3DSocket Reader #1 for port 35815, state=3DRUNNABLE, group=3D= TGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/sun.nio.ch.IOU= til.write1(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelec= torImpl.wakeup(EPollSelectorImpl.java:254) at java.base@11.0.4/java= .nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213) = at java.base@11.0.4/java.nio.channels.spi.AbstractSelector.begin(Abs= tractSelector.java:219) at java.base@11.0.4/sun.nio.ch.SelectorImpl= .begin(SelectorImpl.java:94) at java.base@11.0.4/sun.nio.ch.EPollSe= lectorImpl.doSelect(EPollSelectorImpl.java:116) at java.base@11.0.4= /sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at = java.base@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) = at app//org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.j= ava:1097) at app//org.apache.hadoop.ipc.Server$Listener$Reader.run(= Server.java:1076) 2) Thread[id=3D3499, name=3Dqtp1258237229-3499, state= =3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistributedZkTest] at java.= base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method) at java.ba= se@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java= :234) at java.base@11.0.4/java.util.concurrent.locks.AbstractQueued= Synchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:212= 3) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingA= rrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThr= eadPool$Runner.idleJobPoll(QueuedThreadPool.java:850) at app//org.e= clipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:= 889) at java.base@11.0.4/java.lang.Thread.run(Thread.java:834) 3= ) Thread[id=3D3496, name=3Dqtp1258237229-3496, state=3DRUNNABLE, group=3DTG= RP-HdfsBasicDistributedZkTest] at java.base@11.0.4/sun.nio.ch.EPoll= .wait(Native Method) at java.base@11.0.4/sun.nio.ch.EPollSelectorIm= pl.doSelect(EPollSelectorImpl.java:120) at java.base@11.0.4/sun.nio= .ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.bas= e@11.0.4/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at a= pp//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSel= ector.java:464) at app//org.eclipse.jetty.io.ManagedSelector$Select= orProducer.produce(ManagedSelector.java:401) at app//org.eclipse.je= tty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357= ) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doP= roduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thre= ad.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at a= pp//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYo= uKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambd= a$201/0x0000000100482440.run(Unknown Source) at app//org.eclipse.je= tty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) = at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThr= eadPool.java:917) at java.base@11.0.4/java.lang.Thread.run(Thread.j= ava:834) 4) Thread[id=3D3513, name=3DIPC Server idle connection scanner = for port 35815, state=3DWAITING, group=3DTGRP-HdfsBasicDistributedZkTest] = at java.base@11.0.4/java.lang.Object.wait(Native Method) at = java.base@11.0.4/java.lang.Object.wait(Object.java:328) at java.bas= e@11.0.4/java.util.TimerThread.mainLoop(Timer.java:527) at java.bas= e@11.0.4/java.util.TimerThread.run(Timer.java:506) 5) Thread[id=3D3500, = name=3Dqtp1258237229-3500, state=3DTIMED_WAITING, group=3DTGRP-HdfsBasicDis= tributedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(N= ative Method) at java.base@11.0.4/java.util.concurrent.locks.LockSu= pport.parkNanos(LockSupport.java:234) at java.base@11.0.4/java.util= .concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abs= tractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.B= lockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.ecl= ipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool= .java:850) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$R= unner.run(QueuedThreadPool.java:889) at java.base@11.0.4/java.lang.= Thread.run(Thread.java:834) 6) Thread[id=3D3493, name=3DTimer for 'NameN= ode' metrics system, state=3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistribut= edZkTest] at java.base@11.0.4/java.lang.Object.wait(Native Method) = at java.base@11.0.4/java.util.TimerThread.mainLoop(Timer.java:553) = at java.base@11.0.4/java.util.TimerThread.run(Timer.java:506) 7)= Thread[id=3D3501, name=3Dqtp1258237229-3501, state=3DTIMED_WAITING, group= =3DTGRP-HdfsBasicDistributedZkTest] at java.base@11.0.4/jdk.interna= l.misc.Unsafe.park(Native Method) at java.base@11.0.4/java.util.con= current.locks.LockSupport.parkNanos(LockSupport.java:234) at java.b= ase@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.= eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) = at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobP= oll(QueuedThreadPool.java:850) at app//org.eclipse.jetty.util.threa= d.QueuedThreadPool$Runner.run(QueuedThreadPool.java:889) at java.ba= se@11.0.4/java.lang.Thread.run(Thread.java:834) 8) Thread[id=3D3503, nam= e=3Dqtp1258237229-3503, state=3DTIMED_WAITING, group=3DTGRP-HdfsBasicDistri= butedZkTest] at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Nati= ve Method) at java.base@11.0.4/java.util.concurrent.locks.LockSuppo= rt.parkNanos(LockSupport.java:234) at java.base@11.0.4/java.util.co= ncurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Abstra= ctQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.Bloc= kingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclips= e.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.ja= va:850) a [...truncated too long message...] > at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$160/0x00= 0000010037fc40.run(Unknown Source) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool.runJob(QueuedThreadPool.java:781) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.run(QueuedThreadPool.java:917) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 4) Thread[id=3D9604, name=3Dqtp593746108-9604, state=3D= TIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/jdk.internal.misc.Unsafe.park(= Native Method) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Loc= kSupport.parkNanos(LockSupport.java:234) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Abs= tractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroniz= er.java:2123) [junit4] > at app//org.eclipse.jetty.util.BlockingArrayQueue.= poll(BlockingArrayQueue.java:392) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.idleJobPoll(QueuedThreadPool.java:850) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.run(QueuedThreadPool.java:889) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 5) Thread[id=3D9608, name=3Dqtp593746108-9608, state=3D= TIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/jdk.internal.misc.Unsafe.park(= Native Method) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Loc= kSupport.parkNanos(LockSupport.java:234) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Abs= tractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroniz= er.java:2123) [junit4] > at app//org.eclipse.jetty.util.BlockingArrayQueue.= poll(BlockingArrayQueue.java:392) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.idleJobPoll(QueuedThreadPool.java:850) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.run(QueuedThreadPool.java:889) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 6) Thread[id=3D9609, name=3Dqtp593746108-9609, state=3D= TIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/jdk.internal.misc.Unsafe.park(= Native Method) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Loc= kSupport.parkNanos(LockSupport.java:234) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Abs= tractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroniz= er.java:2123) [junit4] > at app//org.eclipse.jetty.util.BlockingArrayQueue.= poll(BlockingArrayQueue.java:392) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.idleJobPoll(QueuedThreadPool.java:850) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.run(QueuedThreadPool.java:889) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 7) Thread[id=3D9607, name=3Dqtp593746108-9607, state=3D= TIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/jdk.internal.misc.Unsafe.park(= Native Method) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Loc= kSupport.parkNanos(LockSupport.java:234) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Abs= tractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroniz= er.java:2123) [junit4] > at app//org.eclipse.jetty.util.BlockingArrayQueue.= poll(BlockingArrayQueue.java:392) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.idleJobPoll(QueuedThreadPool.java:850) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.run(QueuedThreadPool.java:889) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 8) Thread[id=3D9599, name=3DTimer for 'NameNode' metric= s system, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest= ] [junit4] > at java.base@11.0.4/java.lang.Object.wait(Native M= ethod) [junit4] > at java.base@11.0.4/java.util.TimerThread.mainLoop= (Timer.java:553) [junit4] > at java.base@11.0.4/java.util.TimerThread.run(Time= r.java:506) [junit4] > 9) Thread[id=3D9618, name=3DSocket Reader #1 for port 4= 3727, state=3DRUNNABLE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/java.lang.Thread.holdsLock(Nat= ive Method) [junit4] > at java.base@11.0.4/sun.nio.ch.SelectorImpl.proces= sDeregisterQueue(SelectorImpl.java:245) [junit4] > at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.d= oSelect(EPollSelectorImpl.java:136) [junit4] > at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAn= dDoSelect(SelectorImpl.java:124) [junit4] > at java.base@11.0.4/sun.nio.ch.SelectorImpl.select= (SelectorImpl.java:141) [junit4] > at app//org.apache.hadoop.ipc.Server$Listener$Read= er.doRunLoop(Server.java:1097) [junit4] > at app//org.apache.hadoop.ipc.Server$Listener$Read= er.run(Server.java:1076) [junit4] > 10) Thread[id=3D9603, name=3Dqtp593746108-9603, state=3D= RUNNABLE, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/sun.nio.ch.EPoll.wait(Native M= ethod) [junit4] > at java.base@11.0.4/sun.nio.ch.EPollSelectorImpl.d= oSelect(EPollSelectorImpl.java:120) [junit4] > at java.base@11.0.4/sun.nio.ch.SelectorImpl.lockAn= dDoSelect(SelectorImpl.java:124) [junit4] > at java.base@11.0.4/sun.nio.ch.SelectorImpl.select= (SelectorImpl.java:141) [junit4] > at app//org.eclipse.jetty.io.ManagedSelector$Selec= torProducer.select(ManagedSelector.java:464) [junit4] > at app//org.eclipse.jetty.io.ManagedSelector$Selec= torProducer.produce(ManagedSelector.java:401) [junit4] > at app//org.eclipse.jetty.util.thread.strategy.Eat= WhatYouKill.produceTask(EatWhatYouKill.java:357) [junit4] > at app//org.eclipse.jetty.util.thread.strategy.Eat= WhatYouKill.doProduce(EatWhatYouKill.java:181) [junit4] > at app//org.eclipse.jetty.util.thread.strategy.Eat= WhatYouKill.tryProduce(EatWhatYouKill.java:168) [junit4] > at app//org.eclipse.jetty.util.thread.strategy.Eat= WhatYouKill.produce(EatWhatYouKill.java:132) [junit4] > at app//org.eclipse.jetty.io.ManagedSelector$$Lamb= da$160/0x000000010037fc40.run(Unknown Source) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool.runJob(QueuedThreadPool.java:781) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.run(QueuedThreadPool.java:917) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 11) Thread[id=3D9610, name=3DSession-HouseKeeper-47c0f95= 5, state=3DTIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/jdk.internal.misc.Unsafe.park(= Native Method) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Loc= kSupport.parkNanos(LockSupport.java:234) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Abs= tractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroniz= er.java:2123) [junit4] > at java.base@11.0.4/java.util.concurrent.Scheduled= ThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1= 182) [junit4] > at java.base@11.0.4/java.util.concurrent.Scheduled= ThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:8= 99) [junit4] > at java.base@11.0.4/java.util.concurrent.ThreadPoo= lExecutor.getTask(ThreadPoolExecutor.java:1054) [junit4] > at java.base@11.0.4/java.util.concurrent.ThreadPoo= lExecutor.runWorker(ThreadPoolExecutor.java:1114) [junit4] > at java.base@11.0.4/java.util.concurrent.ThreadPoo= lExecutor$Worker.run(ThreadPoolExecutor.java:628) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 12) Thread[id=3D9606, name=3Dqtp593746108-9606, state=3D= TIMED_WAITING, group=3DTGRP-HdfsChaosMonkeySafeLeaderTest] [junit4] > at java.base@11.0.4/jdk.internal.misc.Unsafe.park(= Native Method) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Loc= kSupport.parkNanos(LockSupport.java:234) [junit4] > at java.base@11.0.4/java.util.concurrent.locks.Abs= tractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchroniz= er.java:2123) [junit4] > at app//org.eclipse.jetty.util.BlockingArrayQueue.= poll(BlockingArrayQueue.java:392) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.idleJobPoll(QueuedThreadPool.java:850) [junit4] > at app//org.eclipse.jetty.util.thread.QueuedThread= Pool$Runner.run(QueuedThreadPool.java:889) [junit4] > at java.base@11.0.4/java.lang.Thread.run(Thread.ja= va:834) [junit4] > 13) Thread[id=3D9619, name=3DIPC Server idle connection = scanner for port 43727, state=3DWAITING, group=3DTGRP-HdfsChaosMonkeySafeLe= aderTest] [junit4] > at java.base@11.0.4/java.lang.Object.wait(Native M= ethod) [junit4] > at java.base@11.0.4/java.lang.Object.wait(Object.j= ava:328) [junit4] > at java.base@11.0.4/java.util.TimerThread.mainLoop= (Timer.java:527) [junit4] > at java.base@11.0.4/java.util.TimerThread.run(Time= r.java:506) [junit4] > =09at __randomizedtesting.SeedInfo.seed([BA0A13FA8B8E0CDD]= :0) [junit4] Completed [804/890 (3!)] on J2 in 13.94s, 0 tests, 3 errors <<<= FAILURES! [...truncated 53132 lines...] ------=_Part_16797_2061718560.1575715553818 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org For additional commands, e-mail: builds-help@lucene.apache.org ------=_Part_16797_2061718560.1575715553818--