Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A4513200BDF for ; Sun, 18 Dec 2016 19:25:59 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id A2DE5160B30; Sun, 18 Dec 2016 18:25:59 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0DB1D160AF6 for ; Sun, 18 Dec 2016 19:25:56 +0100 (CET) Received: (qmail 68989 invoked by uid 500); 18 Dec 2016 18:25:51 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 68975 invoked by uid 99); 18 Dec 2016 18:25:50 -0000 Received: from crius.apache.org (HELO crius.apache.org) (140.211.11.14) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Dec 2016 18:25:50 +0000 Received: from crius.apache.org (localhost [127.0.0.1]) by crius.apache.org (ASF Mail Server at crius.apache.org) with ESMTP id 02C4B9C0189 for ; Sun, 18 Dec 2016 18:25:49 +0000 (UTC) Date: Sun, 18 Dec 2016 18:22:45 +0000 (UTC) From: Apache Jenkins Server To: dev@lucene.apache.org Message-ID: <1646551858.1499.1482085550012.JavaMail.jenkins@crius> In-Reply-To: <109120488.919.1481991815981.JavaMail.jenkins@crius> References: <109120488.919.1481991815981.JavaMail.jenkins@crius> Subject: [JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 231 - Failure MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_1498_293239069.1482085549772" X-Jenkins-Job: Lucene-Solr-NightlyTests-6.x X-Jenkins-Result: FAILURE archived-at: Sun, 18 Dec 2016 18:25:59 -0000 ------=_Part_1498_293239069.1482085549772 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/231/ 7 tests failed. FAILED: org.apache.solr.cloud.CdcrBootstrapTest.testBootstrapWithSourceClu= ster Error Message: Captured an uncaught exception in thread: Thread[id=3D3440, name=3DThread-1= 539, state=3DRUNNABLE, group=3DTGRP-CdcrBootstrapTest] Stack Trace: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an unca= ught exception in thread: Thread[id=3D3440, name=3DThread-1539, state=3DRUN= NABLE, group=3DTGRP-CdcrBootstrapTest] =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740:9C82A20ADBE5940A]= :0) Caused by: java.lang.AssertionError: 1 =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]:0) =09at org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFa= ctory.java:191) =09at org.apache.solr.core.SolrCore.close(SolrCore.java:1360) =09at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:70= 5) =09at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:945) =09at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher= .java:765) =09at java.lang.Thread.run(Thread.java:745) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest Error Message: ObjectTracker found 3 object(s) that were not released!!! [MockDirectoryWra= pper, SolrCore, MockDirectoryWrapper] org.apache.solr.common.util.ObjectRel= easeTracker$ObjectTrackerException at org.apache.solr.common.util.ObjectRe= leaseTracker.track(ObjectReleaseTracker.java:43) at org.apache.solr.core.C= achingDirectoryFactory.get(CachingDirectoryFactory.java:347) at org.apache= .solr.update.SolrIndexWriter.create(SolrIndexWriter.java:66) at org.apache= .solr.core.SolrCore.initIndex(SolrCore.java:672) at org.apache.solr.core.S= olrCore.(SolrCore.java:846) at org.apache.solr.core.SolrCore.(= SolrCore.java:774) at org.apache.solr.core.CoreContainer.create(CoreContai= ner.java:842) at org.apache.solr.core.CoreContainer.create(CoreContainer.j= ava:779) at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static= $0(CoreAdminOperation.java:88) at org.apache.solr.handler.admin.CoreAdminO= peration.execute(CoreAdminOperation.java:377) at org.apache.solr.handler.a= dmin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:365) at org.apac= he.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.j= ava:156) at org.apache.solr.handler.RequestHandlerBase.handleRequest(Reque= stHandlerBase.java:152) at org.apache.solr.servlet.HttpSolrCall.handleAdmi= nRequest(HttpSolrCall.java:664) at org.apache.solr.servlet.HttpSolrCall.ca= ll(HttpSolrCall.java:445) at org.apache.solr.servlet.SolrDispatchFilter.do= Filter(SolrDispatchFilter.java:318) at org.apache.solr.servlet.SolrDispatc= hFilter.doFilter(SolrDispatchFilter.java:269) at org.eclipse.jetty.servlet= .ServletHandler$CachedChain.doFilter(ServletHandler.java:1699) at org.apac= he.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySol= rRunner.java:110) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.= doFilter(ServletHandler.java:1699) at org.eclipse.jetty.servlet.ServletHan= dler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.session= .SessionHandler.doHandle(SessionHandler.java:224) at org.eclipse.jetty.ser= ver.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclip= se.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.ec= lipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) = at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.= java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped= Handler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.han= dle(GzipHandler.java:395) at org.eclipse.jetty.server.handler.HandlerWrapp= er.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.hand= le(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpCha= nnel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpC= onnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallbac= k.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInter= est.fillable(FillInterest.java:95) at org.eclipse.jetty.io.ssl.SslConnecti= on.onFillable(SslConnection.java:202) at org.eclipse.jetty.io.AbstractConn= ection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.= jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.= io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclip= se.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(E= xecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.E= xecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org= .eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduc= eConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJo= b(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThread= Pool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:= 745) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerExcepti= on at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectRelease= Tracker.java:43) at org.apache.solr.core.SolrCore.(SolrCore.java:936= ) at org.apache.solr.core.SolrCore.reload(SolrCore.java:589) at org.apach= e.solr.core.CoreContainer.reload(CoreContainer.java:944) at org.apache.sol= r.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:765) at java.= lang.Thread.run(Thread.java:745) org.apache.solr.common.util.ObjectRelease= Tracker$ObjectTrackerException at org.apache.solr.common.util.ObjectReleas= eTracker.track(ObjectReleaseTracker.java:43) at org.apache.solr.core.Cachi= ngDirectoryFactory.get(CachingDirectoryFactory.java:347) at org.apache.sol= r.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:369) at org.apac= he.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251) at or= g.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:39= 7) at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(Cd= crRequestHandler.java:757) at org.apache.solr.handler.CdcrRequestHandler$B= ootstrapCallable.call(CdcrRequestHandler.java:712) at java.util.concurrent= .FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.Execut= orUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) = at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav= a:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx= ecutor.java:617) at java.lang.Thread.run(Thread.java:745) =20 Stack Trace: java.lang.AssertionError: ObjectTracker found 3 object(s) that were not rel= eased!!! [MockDirectoryWrapper, SolrCore, MockDirectoryWrapper] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException =09at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseT= racker.java:43) =09at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFact= ory.java:347) =09at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:66= ) =09at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:672) =09at org.apache.solr.core.SolrCore.(SolrCore.java:846) =09at org.apache.solr.core.SolrCore.(SolrCore.java:774) =09at org.apache.solr.core.CoreContainer.create(CoreContainer.java:842) =09at org.apache.solr.core.CoreContainer.create(CoreContainer.java:779) =09at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(Core= AdminOperation.java:88) =09at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOpe= ration.java:377) =09at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdmi= nHandler.java:365) =09at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(Core= AdminHandler.java:156) =09at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandl= erBase.java:152) =09at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.= java:664) =09at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445) =09at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte= r.java:318) =09at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilte= r.java:269) =09at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet= Handler.java:1699) =09at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doF= ilter(JettySolrRunner.java:110) =09at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(Servlet= Handler.java:1699) =09at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java= :582) =09at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl= er.java:224) =09at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandl= er.java:1180) =09at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:= 512) =09at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle= r.java:185) =09at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle= r.java:1112) =09at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j= ava:141) =09at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.= java:395) =09at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper= .java:134) =09at org.eclipse.jetty.server.Server.handle(Server.java:534) =09at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) =09at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.jav= a:251) =09at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(Abstra= ctConnection.java:273) =09at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) =09at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:= 202) =09at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(Abstra= ctConnection.java:273) =09at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) =09at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoin= t.java:93) =09at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeP= roduceConsume(ExecuteProduceConsume.java:303) =09at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceC= onsume(ExecuteProduceConsume.java:148) =09at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(Exec= uteProduceConsume.java:136) =09at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo= l.java:671) =09at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool= .java:589) =09at java.lang.Thread.run(Thread.java:745) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException =09at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseT= racker.java:43) =09at org.apache.solr.core.SolrCore.(SolrCore.java:936) =09at org.apache.solr.core.SolrCore.reload(SolrCore.java:589) =09at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:944) =09at org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher= .java:765) =09at java.lang.Thread.run(Thread.java:745) org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException =09at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseT= racker.java:43) =09at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFact= ory.java:347) =09at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.ja= va:369) =09at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.ja= va:251) =09at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler= .java:397) =09at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(Cdc= rRequestHandler.java:757) =09at org.apache.solr.handler.CdcrRequestHandler$BootstrapCallable.call(Cdc= rRequestHandler.java:712) =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.l= ambda$execute$0(ExecutorUtil.java:229) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) =09at java.lang.Thread.run(Thread.java:745) =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]:0) =09at org.junit.Assert.fail(Assert.java:93) =09at org.junit.Assert.assertTrue(Assert.java:43) =09at org.junit.Assert.assertNull(Assert.java:551) =09at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:= 266) =09at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1713) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:870) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at java.lang.Thread.run(Thread.java:745) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDis= tributedZkTest Error Message: ObjectTracker found 1 object(s) that were not released!!! [SolrZkClient] or= g.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException at o= rg.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.= java:43) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.= java:184) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient= .java:116) at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClien= t.java:106) at org.apache.solr.common.cloud.ZkStateReader.(ZkStateRe= ader.java:226) at org.apache.solr.client.solrj.impl.ZkClientClusterStatePr= ovider.connect(ZkClientClusterStateProvider.java:121) at org.apache.solr.c= lient.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:610) at org.= apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(= CloudSolrClient.java:1048) at org.apache.solr.client.solrj.impl.CloudSolrC= lient.request(CloudSolrClient.java:1037) at org.apache.solr.client.solrj.S= olrClient.request(SolrClient.java:1219) at org.apache.solr.handler.CdcrRep= licatorManager.getCheckpoint(CdcrReplicatorManager.java:196) at org.apache= .solr.handler.CdcrReplicatorManager.initLogReaders(CdcrReplicatorManager.ja= va:159) at org.apache.solr.handler.CdcrReplicatorManager.stateUpdate(CdcrR= eplicatorManager.java:134) at org.apache.solr.handler.CdcrStateManager.cal= lback(CdcrStateManager.java:36) at org.apache.solr.handler.CdcrLeaderState= Manager.setAmILeader(CdcrLeaderStateManager.java:108) at org.apache.solr.h= andler.CdcrLeaderStateManager.checkIfIAmLeader(CdcrLeaderStateManager.java:= 95) at org.apache.solr.handler.CdcrLeaderStateManager.access$400(CdcrLeade= rStateManager.java:40) at org.apache.solr.handler.CdcrLeaderStateManager$L= eaderStateWatcher.process(CdcrLeaderStateManager.java:150) at org.apache.s= olr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:268) at= java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) a= t java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.s= olr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(Ex= ecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(= ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$W= orker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java= :745) =20 Stack Trace: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not rel= eased!!! [SolrZkClient] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException =09at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseT= racker.java:43) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:18= 4) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:11= 6) =09at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:10= 6) =09at org.apache.solr.common.cloud.ZkStateReader.(ZkStateReader.java:= 226) =09at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connec= t(ZkClientClusterStateProvider.java:121) =09at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrCl= ient.java:610) =09at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnS= taleState(CloudSolrClient.java:1048) =09at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrCl= ient.java:1037) =09at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) =09at org.apache.solr.handler.CdcrReplicatorManager.getCheckpoint(CdcrRepli= catorManager.java:196) =09at org.apache.solr.handler.CdcrReplicatorManager.initLogReaders(CdcrRepl= icatorManager.java:159) =09at org.apache.solr.handler.CdcrReplicatorManager.stateUpdate(CdcrReplica= torManager.java:134) =09at org.apache.solr.handler.CdcrStateManager.callback(CdcrStateManager.ja= va:36) =09at org.apache.solr.handler.CdcrLeaderStateManager.setAmILeader(CdcrLeade= rStateManager.java:108) =09at org.apache.solr.handler.CdcrLeaderStateManager.checkIfIAmLeader(CdcrL= eaderStateManager.java:95) =09at org.apache.solr.handler.CdcrLeaderStateManager.access$400(CdcrLeaderS= tateManager.java:40) =09at org.apache.solr.handler.CdcrLeaderStateManager$LeaderStateWatcher.pro= cess(CdcrLeaderStateManager.java:150) =09at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkCl= ient.java:268) =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51= 1) =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.l= ambda$execute$0(ExecutorUtil.java:229) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) =09at java.lang.Thread.run(Thread.java:745) =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]:0) =09at org.junit.Assert.fail(Assert.java:93) =09at org.junit.Assert.assertTrue(Assert.java:43) =09at org.junit.Assert.assertNull(Assert.java:551) =09at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:= 266) =09at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1713) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:870) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at java.lang.Thread.run(Thread.java:745) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDis= tributedZkTest Error Message: 2 threads leaked from SUITE scope at org.apache.solr.cloud.CdcrReplicationD= istributedZkTest: 1) Thread[id=3D13299, name=3DzkCallback-2136-thread-5= -processing-n:127.0.0.1:49190_m_sbe%2Fus-EventThread, state=3DWAITING, grou= p=3DTGRP-CdcrReplicationDistributedZkTest] at sun.misc.Unsafe.park(= Native Method) at java.util.concurrent.locks.LockSupport.park(LockS= upport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) = at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:4= 42) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.j= ava:494) 2) Thread[id=3D13298, name=3DzkCallback-2136-thread-5-processin= g-n:127.0.0.1:49190_m_sbe%2Fus-SendThread(127.0.0.1:52678), state=3DTIMED_W= AITING, group=3DTGRP-CdcrReplicationDistributedZkTest] at java.lang= .Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticH= ostProvider.next(StaticHostProvider.java:101) at org.apache.zookeep= er.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.a= pache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from S= UITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:=20 1) Thread[id=3D13299, name=3DzkCallback-2136-thread-5-processing-n:127.0= .0.1:49190_m_sbe%2Fus-EventThread, state=3DWAITING, group=3DTGRP-CdcrReplic= ationDistributedZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175= ) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueu= e.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:= 494) 2) Thread[id=3D13298, name=3DzkCallback-2136-thread-5-processing-n:127.0= .0.1:49190_m_sbe%2Fus-SendThread(127.0.0.1:52678), state=3DTIMED_WAITING, g= roup=3DTGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostPr= ovider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCn= xn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1= 003) =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.CdcrReplicationDis= tributedZkTest Error Message: There are still zombie threads that couldn't be terminated: 1) Thread[id= =3D13298, name=3DzkCallback-2136-thread-5-processing-n:127.0.0.1:49190_m_sb= e%2Fus-SendThread(127.0.0.1:52678), state=3DTIMED_WAITING, group=3DTGRP-Cdc= rReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Met= hod) at org.apache.zookeeper.client.StaticHostProvider.next(StaticH= ostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread= .startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCn= xn$SendThread.run(ClientCnxn.java:1003) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie = threads that couldn't be terminated: 1) Thread[id=3D13298, name=3DzkCallback-2136-thread-5-processing-n:127.0= .0.1:49190_m_sbe%2Fus-SendThread(127.0.0.1:52678), state=3DTIMED_WAITING, g= roup=3DTGRP-CdcrReplicationDistributedZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostPr= ovider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCn= xn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1= 003) =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]:0) FAILED: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.tes= tCollectionsAPI Error Message: Expected to see collection awhollynewcollection_0 null Last available state= : DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/= state.json/5)=3D{ "replicationFactor":"4", "shards":{"shard1":{ "= range":"80000000-7fffffff", "state":"active", "replicas":{}}}, = "router":{"name":"compositeId"}, "maxShardsPerNode":"2", "autoAddRepl= icas":"false"} Stack Trace: java.lang.AssertionError: Expected to see collection awhollynewcollection_0 null Last available state: DocCollection(awhollynewcollection_0//collections/awh= ollynewcollection_0/state.json/5)=3D{ "replicationFactor":"4", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"2", "autoAddReplicas":"false"} =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740:DA1877ADEB2A8D5]:= 0) =09at org.junit.Assert.fail(Assert.java:93) =09at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCas= e.java:237) =09at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollections= API(CollectionsAPIDistributedZkTest.java:496) =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:62) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1713) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Random= izedRunner.java:907) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(Random= izedRunner.java:943) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(Rando= mizedRunner.java:957) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRu= leSetupTeardownChained.java:49) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleT= hreadAndTestName.java:48) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTa= sk(ThreadLeakControl.java:811) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(Threa= dLeakControl.java:462) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran= domizedRunner.java:916) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Random= izedRunner.java:802) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random= izedRunner.java:852) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:863) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at java.lang.Thread.run(Thread.java:745) FAILED: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.tes= tSpecificConfigsets Error Message: KeeperErrorCode =3D NoNode for /collections/withconfigset2 Stack Trace: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =3D N= oNode for /collections/withconfigset2 =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740:68AABC942FA53D4C]= :0) =09at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) =09at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) =09at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java= :356) =09at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java= :353) =09at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecut= or.java:60) =09at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:3= 53) =09at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSpecificCon= figsets(CollectionsAPIDistributedZkTest.java:425) =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:62) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1713) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Random= izedRunner.java:907) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(Random= izedRunner.java:943) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(Rando= mizedRunner.java:957) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRu= leSetupTeardownChained.java:49) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleT= hreadAndTestName.java:48) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTa= sk(ThreadLeakControl.java:811) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(Threa= dLeakControl.java:462) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran= domizedRunner.java:916) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Random= izedRunner.java:802) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random= izedRunner.java:852) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:863) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 11339 lines...] [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributed= ZkTest [junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Luce= ne-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cl= oud.hdfs.HdfsCollectionsAPIDistributedZkTest_45D4F3CED8818740-001/init-core= -data-001 [junit4] 2> 23197 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (fa= lse) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason= =3D, value=3DNaN, ssl=3DNaN, clientAuth=3DNaN) [junit4] 2> 23201 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting c= luster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Nigh= tlyTests-6.x/checkout/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.Hdf= sCollectionsAPIDistributedZkTest_45D4F3CED8818740-001/tempDir-001 [junit4] 2> 23218 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST S= ERVER [junit4] 2> 23227 INFO (Thread-50) [ ] o.a.s.c.ZkTestServer client= port:0.0.0.0/0.0.0.0:0 [junit4] 2> 23228 INFO (Thread-50) [ ] o.a.s.c.ZkTestServer Starti= ng server [junit4] 2> 23327 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.s.c.ZkTestServer start zk server on= port:38251 [junit4] 2> 23482 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ = ] o.a.z.s.NIOServerCnxn Exception causing close of session 0x0 due to java.= io.IOException: ZooKeeperServer not running [junit4] 2> 25253 INFO (jetty-launcher-11-thread-3) [ ] o.e.j.s.Se= rver jetty-9.3.14.v20161028 [junit4] 2> 25254 INFO (jetty-launcher-11-thread-1) [ ] o.e.j.s.Se= rver jetty-9.3.14.v20161028 [junit4] 2> 25262 INFO (jetty-launcher-11-thread-4) [ ] o.e.j.s.Se= rver jetty-9.3.14.v20161028 [junit4] 2> 25254 INFO (jetty-launcher-11-thread-2) [ ] o.e.j.s.Se= rver jetty-9.3.14.v20161028 [junit4] 2> 25329 INFO (jetty-launcher-11-thread-3) [ ] o.e.j.s.h.= ContextHandler Started o.e.j.s.ServletContextHandler@271c81fc{/solr,null,AV= AILABLE} [junit4] 2> 25334 INFO (jetty-launcher-11-thread-3) [ ] o.e.j.s.Ab= stractConnector Started ServerConnector@70838164{HTTP/1.1,[http/1.1]}{127.0= .0.1:55784} [junit4] 2> 25334 INFO (jetty-launcher-11-thread-3) [ ] o.e.j.s.Se= rver Started @29725ms [junit4] 2> 25334 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.c.s.= e.JettySolrRunner Jetty properties: {hostContext=3D/solr, hostPort=3D55784} [junit4] 2> 25335 ERROR (jetty-launcher-11-thread-3) [ ] o.a.s.s.St= artupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing = or incomplete. [junit4] 2> 25335 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.s.So= lrDispatchFilter ___ _ Welcome to Apache Solr=E2=84=A2 version = 6.4.0 [junit4] 2> 25335 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.s.So= lrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 25335 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.s.So= lrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 25335 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.s.So= lrDispatchFilter |___/\___/_|_| Start time: 2016-12-18T16:54:54.219Z [junit4] 2> 25465 INFO (jetty-launcher-11-thread-2) [ ] o.e.j.s.h.= ContextHandler Started o.e.j.s.ServletContextHandler@77a0a2cd{/solr,null,AV= AILABLE} [junit4] 2> 25475 INFO (jetty-launcher-11-thread-4) [ ] o.e.j.s.h.= ContextHandler Started o.e.j.s.ServletContextHandler@1b66d217{/solr,null,AV= AILABLE} [junit4] 2> 25482 INFO (jetty-launcher-11-thread-1) [ ] o.e.j.s.h.= ContextHandler Started o.e.j.s.ServletContextHandler@3a4dda0b{/solr,null,AV= AILABLE} [junit4] 2> 25498 INFO (jetty-launcher-11-thread-1) [ ] o.e.j.s.Ab= stractConnector Started ServerConnector@6570b9b{HTTP/1.1,[http/1.1]}{127.0.= 0.1:47833} [junit4] 2> 25501 INFO (jetty-launcher-11-thread-1) [ ] o.e.j.s.Se= rver Started @29892ms [junit4] 2> 25498 INFO (jetty-launcher-11-thread-4) [ ] o.e.j.s.Ab= stractConnector Started ServerConnector@6a75f5ac{HTTP/1.1,[http/1.1]}{127.0= .0.1:58398} [junit4] 2> 25501 INFO (jetty-launcher-11-thread-4) [ ] o.e.j.s.Se= rver Started @29892ms [junit4] 2> 25499 INFO (jetty-launcher-11-thread-2) [ ] o.e.j.s.Ab= stractConnector Started ServerConnector@2387bc8e{HTTP/1.1,[http/1.1]}{127.0= .0.1:33161} [junit4] 2> 25501 INFO (jetty-launcher-11-thread-2) [ ] o.e.j.s.Se= rver Started @29892ms [junit4] 2> 25501 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.c.s.= e.JettySolrRunner Jetty properties: {hostContext=3D/solr, hostPort=3D47833} [junit4] 2> 25501 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.c.s.= e.JettySolrRunner Jetty properties: {hostContext=3D/solr, hostPort=3D58398} [junit4] 2> 25501 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.c.s.= e.JettySolrRunner Jetty properties: {hostContext=3D/solr, hostPort=3D33161} [junit4] 2> 25504 ERROR (jetty-launcher-11-thread-1) [ ] o.a.s.s.St= artupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing = or incomplete. [junit4] 2> 25505 ERROR (jetty-launcher-11-thread-4) [ ] o.a.s.s.St= artupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing = or incomplete. [junit4] 2> 25505 ERROR (jetty-launcher-11-thread-2) [ ] o.a.s.s.St= artupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing = or incomplete. [junit4] 2> 25506 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.s.So= lrDispatchFilter ___ _ Welcome to Apache Solr=E2=84=A2 version = 6.4.0 [junit4] 2> 25507 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.s.So= lrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 25507 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.s.So= lrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 25507 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.s.So= lrDispatchFilter |___/\___/_|_| Start time: 2016-12-18T16:54:54.391Z [junit4] 2> 25506 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.s.So= lrDispatchFilter ___ _ Welcome to Apache Solr=E2=84=A2 version = 6.4.0 [junit4] 2> 25508 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.s.So= lrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 25508 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.s.So= lrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 25508 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.s.So= lrDispatchFilter |___/\___/_|_| Start time: 2016-12-18T16:54:54.392Z [junit4] 2> 25506 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.s.So= lrDispatchFilter ___ _ Welcome to Apache Solr=E2=84=A2 version = 6.4.0 [junit4] 2> 25508 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.s.So= lrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 25509 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.s.So= lrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 25509 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.s.So= lrDispatchFilter |___/\___/_|_| Start time: 2016-12-18T16:54:54.393Z [junit4] 2> 25556 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.s.So= lrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 25579 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ = ] o.a.z.s.NIOServerCnxn caught end of stream exception [junit4] 2> EndOfStreamException: Unable to read additional data from = client sessionid 0x15912db80410001, likely client has closed socket [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOSe= rverCnxn.java:228) [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxnFactory.run= (NIOServerCnxnFactory.java:208) [junit4] 2> =09at java.lang.Thread.run(Thread.java:745) [junit4] 2> 25677 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.s.So= lrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 25694 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.s.So= lrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 25701 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.s.So= lrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 25761 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.u.Up= dateShardHandler Creating UpdateShardHandler HTTP client with params: socke= tTimeout=3D340000&connTimeout=3D45000&retry=3Dtrue [junit4] 2> 25762 INFO (jetty-launcher-11-thread-2) [ ] o.a.s.c.Zk= Container Zookeeper client=3D127.0.0.1:38251/solr [junit4] 2> 25777 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.u.Up= dateShardHandler Creating UpdateShardHandler HTTP client with params: socke= tTimeout=3D340000&connTimeout=3D45000&retry=3Dtrue [junit4] 2> 25781 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.u.Up= dateShardHandler Creating UpdateShardHandler HTTP client with params: socke= tTimeout=3D340000&connTimeout=3D45000&retry=3Dtrue [junit4] 2> 25793 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.u.Up= dateShardHandler Creating UpdateShardHandler HTTP client with params: socke= tTimeout=3D340000&connTimeout=3D45000&retry=3Dtrue [junit4] 2> 25821 INFO (jetty-launcher-11-thread-1) [ ] o.a.s.c.Zk= Container Zookeeper client=3D127.0.0.1:38251/solr [junit4] 2> 25861 INFO (jetty-launcher-11-thread-4) [ ] o.a.s.c.Zk= Container Zookeeper client=3D127.0.0.1:38251/solr [junit4] 2> 25909 INFO (jetty-launcher-11-thread-3) [ ] o.a.s.c.Zk= Container Zookeeper client=3D127.0.0.1:38251/solr [junit4] 2> 26445 INFO (jetty-launcher-11-thread-2) [n:127.0.0.1:3316= 1_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127= .0.0.1:33161_solr [junit4] 2> 26451 INFO (jetty-launcher-11-thread-1) [n:127.0.0.1:4783= 3_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_n= odes/127.0.0.1:47833_solr [junit4] 2> 26462 INFO (jetty-launcher-11-thread-3) [n:127.0.0.1:5578= 4_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_n= odes/127.0.0.1:55784_solr [junit4] 2> 26462 INFO (jetty-launcher-11-thread-2) [n:127.0.0.1:3316= 1_solr ] o.a.s.c.Overseer Overseer (id=3D97129600924057609-127.0.0.1:331= 61_solr-n_0000000000) starting [junit4] 2> 26494 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (0) -> (1) [junit4] 2> 26494 INFO (zkCallback-29-thread-1-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (0) -> (1) [junit4] 2> 26495 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (0) -> (1) [junit4] 2> 26499 INFO (zkCallback-29-thread-1-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (1) -> (2) [junit4] 2> 26499 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (1) -> (2) [junit4] 2> 26499 INFO (zkCallback-31-thread-1-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (0) -> (1) [junit4] 2> 26500 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (1) -> (2) [junit4] 2> 26599 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (1) -> (2) [junit4] 2> 26609 INFO (jetty-launcher-11-thread-4) [n:127.0.0.1:5839= 8_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_n= odes/127.0.0.1:58398_solr [junit4] 2> 26613 INFO (zkCallback-29-thread-1-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (2) -> (3) [junit4] 2> 26613 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (2) -> (3) [junit4] 2> 26613 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (2) -> (3) [junit4] 2> 26613 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (2) -> (3) [junit4] 2> 26998 INFO (jetty-launcher-11-thread-2) [n:127.0.0.1:3316= 1_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_n= odes/127.0.0.1:33161_solr [junit4] 2> 27003 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (3) -> (4) [junit4] 2> 27004 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (3) -> (4) [junit4] 2> 27004 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (3) -> (4) [junit4] 2> 27008 INFO (zkCallback-29-thread-1-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader Updated l= ive nodes from ZooKeeper... (3) -> (4) [junit4] 2> 27266 INFO (jetty-launcher-11-thread-1) [n:127.0.0.1:4783= 3_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions undernea= th /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkou= t/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistr= ibutedZkTest_45D4F3CED8818740-001/tempDir-001/node1/. [junit4] 2> 27312 INFO (jetty-launcher-11-thread-4) [n:127.0.0.1:5839= 8_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions undernea= th /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkou= t/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistr= ibutedZkTest_45D4F3CED8818740-001/tempDir-001/node4/. [junit4] 2> 27324 INFO (jetty-launcher-11-thread-3) [n:127.0.0.1:5578= 4_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions undernea= th /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkou= t/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistr= ibutedZkTest_45D4F3CED8818740-001/tempDir-001/node3/. [junit4] 2> 27328 INFO (jetty-launcher-11-thread-2) [n:127.0.0.1:3316= 1_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions undernea= th /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkou= t/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistr= ibutedZkTest_45D4F3CED8818740-001/tempDir-001/node2/. [junit4] 2> 27759 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live no= des from ZooKeeper... (0) -> (4) [junit4] 2> 27761 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvide= r Cluster at 127.0.0.1:38251/solr ready [junit4] 2> 29372 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.h.u.NativeCodeLoader Unable to load= native-hadoop library for your platform... using builtin-java classes wher= e applicable [junit4] 1> Formatting using clusterid: testClusterID [junit4] 2> 31280 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate c= onfiguration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.pro= perties [junit4] 2> 31740 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log Logging to org.slf4j.impl.Log4j= LoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog [junit4] 2> 31818 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request lo= g can only be enabled using Log4j [junit4] 2> 31900 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 31992 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.i= vy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/= webapps/hdfs to ./temp/Jetty_localhost_46684_hdfs____hlxk7u/webapp [junit4] 2> 33239 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log Started HttpServer2$SelectChann= elConnectorWithSafeStartup@localhost:46684 [junit4] 2> 34319 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request lo= g can only be enabled using Log4j [junit4] 2> 34325 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 34339 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.i= vy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/= webapps/datanode to ./temp/Jetty_localhost_50271_datanode____m07fqw/webapp [junit4] 2> 34846 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log Started HttpServer2$SelectChann= elConnectorWithSafeStartup@localhost:50271 [junit4] 2> 35865 WARN (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request lo= g can only be enabled using Log4j [junit4] 2> 35867 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 35897 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.i= vy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/= webapps/datanode to ./temp/Jetty_localhost_48080_datanode____krlwjv/webapp [junit4] 2> 36637 INFO (SUITE-HdfsCollectionsAPIDistributedZkTest-see= d#[45D4F3CED8818740]-worker) [ ] o.m.log Started HttpServer2$SelectChann= elConnectorWithSafeStartup@localhost:48080 [junit4] 2> 39623 INFO (IPC Server handler 8 on 56880) [ ] BlockSt= ateChange BLOCK* processReport: from storage DS-580d30a0-75a3-4473-b355-2fd= 923a32259 node DatanodeRegistration(127.0.0.1:56590, datanodeUuid=3Dd4ba5c8= 3-c67c-4fa8-bd15-49b07d24b129, infoPort=3D51310, infoSecurePort=3D0, ipcPor= t=3D42938, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D795121145;c=3D= 0), blocks: 0, hasStaleStorage: true, processing time: 5 msecs [junit4] 2> 39624 INFO (IPC Server handler 8 on 56880) [ ] BlockSt= ateChange BLOCK* processReport: from storage DS-60b8353a-3c36-45cb-8a75-586= 3f703da92 node DatanodeRegistration(127.0.0.1:56590, datanodeUuid=3Dd4ba5c8= 3-c67c-4fa8-bd15-49b07d24b129, infoPort=3D51310, infoSecurePort=3D0, ipcPor= t=3D42938, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D795121145;c=3D= 0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs [junit4] 2> 39625 INFO (IPC Server handler 6 on 56880) [ ] BlockSt= ateChange BLOCK* processReport: from storage DS-2775f86c-8bf9-418d-82aa-ab1= 44e012059 node DatanodeRegistration(127.0.0.1:52097, datanodeUuid=3Dd63b4e3= d-24a9-47a0-a9ce-e54987e5fb7d, infoPort=3D49203, infoSecurePort=3D0, ipcPor= t=3D34154, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D795121145;c=3D= 0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs [junit4] 2> 39626 INFO (IPC Server handler 6 on 56880) [ ] BlockSt= ateChange BLOCK* processReport: from storage DS-b3a07fc5-727b-4d95-ac1f-a8e= c59759fdc node DatanodeRegistration(127.0.0.1:52097, datanodeUuid=3Dd63b4e3= d-24a9-47a0-a9ce-e54987e5fb7d, infoPort=3D49203, infoSecurePort=3D0, ipcPor= t=3D34154, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D795121145;c=3D= 0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs [junit4] 2> 40120 INFO (TEST-HdfsCollectionsAPIDistributedZkTest.test= MaxNodesPerShard-seed#[45D4F3CED8818740]) [ ] o.a.s.SolrTestCaseJ4 ###St= arting testMaxNodesPerShard [junit4] 2> 40122 INFO (TEST-HdfsCollectionsAPIDistributedZkTest.test= MaxNodesPerShard-seed#[45D4F3CED8818740]) [ ] o.a.s.c.c.ZkStateReader Up= dated live nodes from ZooKeeper... (0) -> (4) [junit4] 2> 40170 INFO (qtp1957733841-139) [n:127.0.0.1:33161_solr = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with para= ms replicationFactor=3D2&collection.configName=3Dconf&name=3Doversharded&ac= tion=3DCREATE&numShards=3D3&wt=3Djavabin&version=3D2 and sendToOCPQueue=3Dt= rue [junit4] 2> 40199 INFO (OverseerThreadFactory-84-thread-1-processing-= n:127.0.0.1:33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.CreateCollecti= onCmd Create collection oversharded [junit4] 2> 40204 ERROR (OverseerThreadFactory-84-thread-1-processing-= n:127.0.0.1:33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.OverseerCollec= tionMessageHandler Collection: oversharded operation: create failed:org.apa= che.solr.common.SolrException: Cannot create collection oversharded. Value = of maxShardsPerNode is 1, and the number of nodes currently live or live an= d part of your createNodeSet is 4. This allows a maximum of 4 to be created= . Value of numShards is 3 and value of replicationFactor is 2. This require= s 6 shards to be created (higher than the allowed number) [junit4] 2> =09at org.apache.solr.cloud.CreateCollectionCmd.call(Creat= eCollectionCmd.java:155) [junit4] 2> =09at org.apache.solr.cloud.OverseerCollectionMessageHandl= er.processMessage(OverseerCollectionMessageHandler.java:224) [junit4] 2> =09at org.apache.solr.cloud.OverseerTaskProcessor$Runner.r= un(OverseerTaskProcessor.java:463) [junit4] 2> =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThr= eadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) [junit4] 2> =09at java.util.concurrent.ThreadPoolExecutor.runWorker(Th= readPoolExecutor.java:1142) [junit4] 2> =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(T= hreadPoolExecutor.java:617) [junit4] 2> =09at java.lang.Thread.run(Thread.java:745) [junit4] 2>=20 [junit4] 2> 40264 INFO (qtp1957733841-139) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections par= ams=3D{replicationFactor=3D2&collection.configName=3Dconf&name=3Doversharde= d&action=3DCREATE&numShards=3D3&wt=3Djavabin&version=3D2} status=3D400 QTim= e=3D122 [junit4] 2> 40267 INFO (TEST-HdfsCollectionsAPIDistributedZkTest.test= MaxNodesPerShard-seed#[45D4F3CED8818740]) [ ] o.a.s.SolrTestCaseJ4 ###En= ding testMaxNodesPerShard [junit4] 2> 40330 INFO (TEST-HdfsCollectionsAPIDistributedZkTest.test= CollectionsAPI-seed#[45D4F3CED8818740]) [ ] o.a.s.SolrTestCaseJ4 ###Star= ting testCollectionsAPI [junit4] 2> 40338 INFO (TEST-HdfsCollectionsAPIDistributedZkTest.test= CollectionsAPI-seed#[45D4F3CED8818740]) [ ] o.a.s.c.c.ZkStateReader Upda= ted live nodes from ZooKeeper... (0) -> (4) [junit4] 2> 40344 INFO (qtp1957733841-140) [n:127.0.0.1:33161_solr = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with para= ms async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c&replicationFactor=3D4&colle= ction.configName=3Dconf2&maxShardsPerNode=3D2&name=3Dawhollynewcollection_0= &action=3DCREATE&numShards=3D1&wt=3Djavabin&version=3D2 and sendToOCPQueue= =3Dtrue [junit4] 2> 40347 INFO (qtp1957733841-140) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections par= ams=3D{async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c&replicationFactor=3D4&c= ollection.configName=3Dconf2&maxShardsPerNode=3D2&name=3Dawhollynewcollecti= on_0&action=3DCREATE&numShards=3D1&wt=3Djavabin&version=3D2} status=3D0 QTi= me=3D3 [junit4] 2> 40351 INFO (qtp1957733841-141) [n:127.0.0.1:33161_solr = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with para= ms async=3D7f796b8f-f956-4d36-a92d-f8fd68224824&replicationFactor=3D3&colle= ction.configName=3Dconf2&maxShardsPerNode=3D4&name=3Dawhollynewcollection_1= &action=3DCREATE&numShards=3D4&wt=3Djavabin&version=3D2 and sendToOCPQueue= =3Dtrue [junit4] 2> 40355 INFO (qtp1957733841-141) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections par= ams=3D{async=3D7f796b8f-f956-4d36-a92d-f8fd68224824&replicationFactor=3D3&c= ollection.configName=3Dconf2&maxShardsPerNode=3D4&name=3Dawhollynewcollecti= on_1&action=3DCREATE&numShards=3D4&wt=3Djavabin&version=3D2} status=3D0 QTi= me=3D4 [junit4] 2> 40374 INFO (OverseerCollectionConfigSetProcessor-97129600= 924057609-127.0.0.1:33161_solr-n_0000000000) [n:127.0.0.1:33161_solr ] o= .a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/= qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeep= er [junit4] 2> 40378 INFO (OverseerThreadFactory-84-thread-2-processing-= n:127.0.0.1:33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.CreateCollecti= onCmd Create collection awhollynewcollection_0 [junit4] 2> 40401 INFO (OverseerThreadFactory-84-thread-3-processing-= n:127.0.0.1:33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.CreateCollecti= onCmd Create collection awhollynewcollection_1 [junit4] 2> 40446 INFO (zkCallback-40-thread-3) [ ] o.a.s.c.c.ZkSt= ateReader A cluster state change: [WatchedEvent state:SyncConnected type:No= deCreated path:/collections/awhollynewcollection_0/state.json] for collecti= on [awhollynewcollection_0] has occurred - updating... (live nodes size: [4= ]) [junit4] 2> 40973 INFO (qtp960718685-129) [n:127.0.0.1:55784_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456358956171520&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_0_shard1_replica4&action=3DCREATE&numShards=3D1&collection=3Dawhol= lynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D37 [junit4] 2> 40978 INFO (parallelCoreAdminExecutor-89-thread-1-process= ing-n:127.0.0.1:55784_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589561= 71520 CREATE) [n:127.0.0.1:55784_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456358956171= 520&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_0_shard1_replica4&action=3DCREATE&numShards=3D1&c= ollection=3Dawhollynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2 [junit4] 2> 40982 INFO (qtp1638429046-149) [n:127.0.0.1:58398_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456358941029277&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_0_shard1_replica3&action=3DCREATE&numShards=3D1&collection=3Dawho= llynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D48 [junit4] 2> 40993 INFO (parallelCoreAdminExecutor-87-thread-1-process= ing-n:127.0.0.1:58398_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589410= 29277 CREATE) [n:127.0.0.1:58398_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456358941029= 277&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_0_shard1_replica3&action=3DCREATE&numShards=3D1&c= ollection=3Dawhollynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2 [junit4] 2> 41213 INFO (qtp944827007-157) [n:127.0.0.1:47833_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359452178139&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_1_shard4_replica3&action=3DCREATE&numShards=3D4&collection=3Dawhol= lynewcollection_1&shard=3Dshard4&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D22 [junit4] 2> 41219 INFO (parallelCoreAdminExecutor-85-thread-1-process= ing-n:127.0.0.1:47833_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594521= 78139 CREATE) [n:127.0.0.1:47833_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359452178= 139&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard4_replica3&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard4&wt=3Djavabin&version=3D2 [junit4] 2> 41221 INFO (qtp1957733841-137) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456359340567608&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_0_shard1_replica2&action=3DCREATE&numShards=3D1&collection=3Dawho= llynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D269 [junit4] 2> 41221 INFO (parallelCoreAdminExecutor-91-thread-1-process= ing-n:127.0.0.1:33161_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563593405= 67608 CREATE) [n:127.0.0.1:33161_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456359340567= 608&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_0_shard1_replica2&action=3DCREATE&numShards=3D1&c= ollection=3Dawhollynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2 [junit4] 2> 41237 INFO (qtp944827007-157) [n:127.0.0.1:47833_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359484201352&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_1_shard2_replica1&action=3DCREATE&numShards=3D4&collection=3Dawhol= lynewcollection_1&shard=3Dshard2&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D22 [junit4] 2> 41237 INFO (parallelCoreAdminExecutor-85-thread-2-process= ing-n:127.0.0.1:47833_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594842= 01352 CREATE) [n:127.0.0.1:47833_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359484201= 352&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard2_replica1&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard2&wt=3Djavabin&version=3D2 [junit4] 2> 41306 INFO (zkCallback-40-thread-3) [ ] o.a.s.c.c.ZkSt= ateReader A cluster state change: [WatchedEvent state:SyncConnected type:No= deDataChanged path:/collections/awhollynewcollection_0/state.json] for coll= ection [awhollynewcollection_0] has occurred - updating... (live nodes size= : [4]) [junit4] 2> 41307 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41307 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41307 INFO (zkCallback-29-thread-5-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41323 INFO (qtp1638429046-150) [n:127.0.0.1:58398_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359384672860&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_1_shard1_replica2&action=3DCREATE&numShards=3D4&collection=3Dawho= llynewcollection_1&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D99 [junit4] 2> 41361 INFO (qtp960718685-130) [n:127.0.0.1:55784_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359442398387&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_1_shard3_replica1&action=3DCREATE&numShards=3D4&collection=3Dawhol= lynewcollection_1&shard=3Dshard3&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D166 [junit4] 2> 41362 INFO (parallelCoreAdminExecutor-89-thread-2-process= ing-n:127.0.0.1:55784_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594423= 98387 CREATE) [n:127.0.0.1:55784_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359442398= 387&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard3_replica1&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard3&wt=3Djavabin&version=3D2 [junit4] 2> 41373 INFO (qtp1638429046-147) [n:127.0.0.1:58398_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359463301937&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_1_shard4_replica1&action=3DCREATE&numShards=3D4&collection=3Dawho= llynewcollection_1&shard=3Dshard4&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D185 [junit4] 2> 41405 INFO (qtp1957733841-136) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359496675840&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_1_shard2_replica2&action=3DCREATE&numShards=3D4&collection=3Dawho= llynewcollection_1&shard=3Dshard2&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D96 [junit4] 2> 41409 INFO (parallelCoreAdminExecutor-87-thread-2-process= ing-n:127.0.0.1:58398_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563593846= 72860 CREATE) [n:127.0.0.1:58398_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359384672= 860&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard1_replica2&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard1&wt=3Djavabin&version=3D2 [junit4] 2> 41409 INFO (parallelCoreAdminExecutor-91-thread-2-process= ing-n:127.0.0.1:33161_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594966= 75840 CREATE) [n:127.0.0.1:33161_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359496675= 840&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard2_replica2&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard2&wt=3Djavabin&version=3D2 [junit4] 2> 41410 INFO (parallelCoreAdminExecutor-87-thread-4-process= ing-n:127.0.0.1:58398_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563595041= 52651 CREATE) [n:127.0.0.1:58398_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359504152= 651&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard2_replica3&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard2&wt=3Djavabin&version=3D2 [junit4] 2> 41413 INFO (parallelCoreAdminExecutor-87-thread-3-process= ing-n:127.0.0.1:58398_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594633= 01937 CREATE) [n:127.0.0.1:58398_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359463301= 937&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard4_replica1&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard4&wt=3Djavabin&version=3D2 [junit4] 2> 41417 INFO (qtp1638429046-151) [n:127.0.0.1:58398_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359504152651&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_1_shard2_replica3&action=3DCREATE&numShards=3D4&collection=3Dawho= llynewcollection_1&shard=3Dshard2&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D10 [junit4] 2> 41433 INFO (qtp1957733841-139) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359572617555&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_1_shard3_replica3&action=3DCREATE&numShards=3D4&collection=3Dawho= llynewcollection_1&shard=3Dshard3&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D109 [junit4] 2> 41434 INFO (parallelCoreAdminExecutor-85-thread-3-process= ing-n:127.0.0.1:47833_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594888= 16146 CREATE) [n:127.0.0.1:47833_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359488816= 146&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard3_replica2&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard3&wt=3Djavabin&version=3D2 [junit4] 2> 41437 INFO (qtp944827007-160) [n:127.0.0.1:47833_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359488816146&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_1_shard3_replica2&action=3DCREATE&numShards=3D4&collection=3Dawhol= lynewcollection_1&shard=3Dshard3&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D211 [junit4] 2> 41457 INFO (parallelCoreAdminExecutor-89-thread-4-process= ing-n:127.0.0.1:55784_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594425= 73119 CREATE) [n:127.0.0.1:55784_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359442573= 119&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard4_replica2&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard4&wt=3Djavabin&version=3D2 [junit4] 2> 41501 INFO (parallelCoreAdminExecutor-91-thread-3-process= ing-n:127.0.0.1:33161_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563595726= 17555 CREATE) [n:127.0.0.1:33161_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359572617= 555&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard3_replica3&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard3&wt=3Djavabin&version=3D2 [junit4] 2> 41505 INFO (parallelCoreAdminExecutor-91-thread-4-process= ing-n:127.0.0.1:33161_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563595068= 23806 CREATE) [n:127.0.0.1:33161_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359506823= 806&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard1_replica1&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard1&wt=3Djavabin&version=3D2 [junit4] 2> 41507 INFO (qtp1957733841-140) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359506823806&qt=3D/admin/= cores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewc= ollection_1_shard1_replica1&action=3DCREATE&numShards=3D4&collection=3Dawho= llynewcollection_1&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTim= e=3D176 [junit4] 2> 41507 INFO (parallelCoreAdminExecutor-85-thread-4-process= ing-n:127.0.0.1:47833_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589662= 33972 CREATE) [n:127.0.0.1:47833_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456358966233= 972&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_0_shard1_replica1&action=3DCREATE&numShards=3D1&c= ollection=3Dawhollynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2 [junit4] 2> 41508 INFO (qtp944827007-159) [n:127.0.0.1:47833_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456358966233972&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_0_shard1_replica1&action=3DCREATE&numShards=3D1&collection=3Dawhol= lynewcollection_0&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D134 [junit4] 2> 41509 INFO (parallelCoreAdminExecutor-89-thread-3-process= ing-n:127.0.0.1:55784_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594684= 02965 CREATE) [n:127.0.0.1:55784_solr ] o.a.s.h.a.CoreAdminOperation cor= e create command async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359468402= 965&qt=3D/admin/cores&collection.configName=3Dconf2&newCollection=3Dtrue&na= me=3Dawhollynewcollection_1_shard1_replica3&action=3DCREATE&numShards=3D4&c= ollection=3Dawhollynewcollection_1&shard=3Dshard1&wt=3Djavabin&version=3D2 [junit4] 2> 41510 INFO (qtp960718685-126) [n:127.0.0.1:55784_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359468402965&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_1_shard1_replica3&action=3DCREATE&numShards=3D4&collection=3Dawhol= lynewcollection_1&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D96 [junit4] 2> 41511 INFO (qtp960718685-131) [n:127.0.0.1:55784_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{= async=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359442573119&qt=3D/admin/c= ores&collection.configName=3Dconf2&newCollection=3Dtrue&name=3Dawhollynewco= llection_1_shard4_replica2&action=3DCREATE&numShards=3D4&collection=3Dawhol= lynewcollection_1&shard=3Dshard4&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D96 [junit4] 2> 41520 INFO (qtp1957733841-136) [n:127.0.0.1:33161_solr = ] o.a.s.h.a.CoreAdminOperation Checking request status for : 48e75d5b-d028= -4bae-a0b4-7d4ada0f198c20456359340567608 [junit4] 2> 41520 INFO (qtp1957733841-136) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {qt=3D/admin/cores&requestid=3D48e75d5b-d028-4bae-a0b4-7d4ada0f198c20456359= 340567608&action=3DREQUESTSTATUS&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D0 [junit4] 2> 41521 INFO (qtp1957733841-136) [n:127.0.0.1:33161_solr = ] o.a.s.h.a.CoreAdminOperation Checking request status for : 7f796b8f-f956= -4d36-a92d-f8fd6822482420456359572617555 [junit4] 2> 41521 INFO (qtp1957733841-136) [n:127.0.0.1:33161_solr = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D= {qt=3D/admin/cores&requestid=3D7f796b8f-f956-4d36-a92d-f8fd6822482420456359= 572617555&action=3DREQUESTSTATUS&wt=3Djavabin&version=3D2} status=3D0 QTime= =3D0 [junit4] 2> 41566 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41567 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41567 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41568 INFO (zkCallback-29-thread-5-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41582 INFO (zkCallback-40-thread-3) [ ] o.a.s.c.c.ZkSt= ateReader A cluster state change: [WatchedEvent state:SyncConnected type:No= deDataChanged path:/collections/awhollynewcollection_0/state.json] for coll= ection [awhollynewcollection_0] has occurred - updating... (live nodes size= : [4]) [junit4] 2> 41582 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41583 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41584 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41692 INFO (zkCallback-29-thread-5-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41730 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41732 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41733 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41733 INFO (zkCallback-29-thread-5-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41787 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41787 INFO (zkCallback-40-thread-3) [ ] o.a.s.c.c.ZkSt= ateReader A cluster state change: [WatchedEvent state:SyncConnected type:No= deDataChanged path:/collections/awhollynewcollection_0/state.json] for coll= ection [awhollynewcollection_0] has occurred - updating... (live nodes size= : [4]) [junit4] 2> 41787 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41787 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41789 INFO (zkCallback-29-thread-5-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_0/state.json] for collection [awhollynewc= ollection_0] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41919 INFO (zkCallback-32-thread-1-processing-n:127.0.0.1= :55784_solr) [n:127.0.0.1:55784_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41920 INFO (zkCallback-31-thread-2-processing-n:127.0.0.1= :58398_solr) [n:127.0.0.1:58398_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41920 INFO (zkCallback-30-thread-1-processing-n:127.0.0.1= :47833_solr) [n:127.0.0.1:47833_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 41923 INFO (zkCallback-29-thread-5-processing-n:127.0.0.1= :33161_solr) [n:127.0.0.1:33161_solr ] o.a.s.c.c.ZkStateReader A cluster= state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:= /collections/awhollynewcollection_1/state.json] for collection [awhollynewc= ollection_1] has occurred - updating... (live nodes size: [4]) [junit4] 2> 42254 INFO (parallelCoreAdminExecutor-89-thread-1-process= ing-n:127.0.0.1:55784_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589561= 71520 CREATE) [n:127.0.0.1:55784_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica4] o.a.s.c.SolrConfig Using Lucene Mat= chVersion: 6.4.0 [junit4] 2> 42282 INFO (parallelCoreAdminExecutor-87-thread-1-process= ing-n:127.0.0.1:58398_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589410= 29277 CREATE) [n:127.0.0.1:58398_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica3] o.a.s.c.SolrConfig Using Lucene Mat= chVersion: 6.4.0 [junit4] 2> 42348 INFO (parallelCoreAdminExecutor-89-thread-1-process= ing-n:127.0.0.1:55784_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589561= 71520 CREATE) [n:127.0.0.1:55784_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica4] o.a.s.s.IndexSchema [awhollynewcoll= ection_0_shard1_replica4] Schema name=3Dminimal [junit4] 2> 42377 INFO (parallelCoreAdminExecutor-87-thread-1-process= ing-n:127.0.0.1:58398_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589410= 29277 CREATE) [n:127.0.0.1:58398_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica3] o.a.s.s.IndexSchema [awhollynewcoll= ection_0_shard1_replica3] Schema name=3Dminimal [junit4] 2> 42380 INFO (parallelCoreAdminExecutor-91-thread-1-process= ing-n:127.0.0.1:33161_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563593405= 67608 CREATE) [n:127.0.0.1:33161_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica2] o.a.s.c.SolrConfig Using Lucene Mat= chVersion: 6.4.0 [junit4] 2> 42399 INFO (parallelCoreAdminExecutor-89-thread-1-process= ing-n:127.0.0.1:55784_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589561= 71520 CREATE) [n:127.0.0.1:55784_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica4] o.a.s.s.IndexSchema Loaded schema m= inimal/1.1 with uniqueid field id [junit4] 2> 42399 INFO (parallelCoreAdminExecutor-85-thread-2-process= ing-n:127.0.0.1:47833_solr 7f796b8f-f956-4d36-a92d-f8fd68224824204563594842= 01352 CREATE) [n:127.0.0.1:47833_solr c:awhollynewcollection_1 s:shard2 x:= awhollynewcollection_1_shard2_replica1] o.a.s.c.SolrConfig Using Lucene Mat= chVersion: 6.4.0 [junit4] 2> 42401 INFO (parallelCoreAdminExecutor-89-thread-1-process= ing-n:127.0.0.1:55784_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589561= 71520 CREATE) [n:127.0.0.1:55784_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica4] o.a.s.c.CoreContainer Creating Solr= Core 'awhollynewcollection_0_shard1_replica4' using configuration from coll= ection awhollynewcollection_0 [junit4] 2> 42402 INFO (parallelCoreAdminExecutor-87-thread-1-process= ing-n:127.0.0.1:58398_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589410= 29277 CREATE) [n:127.0.0.1:58398_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica3] o.a.s.s.IndexSchema Loaded schema m= inimal/1.1 with uniqueid field id [junit4] 2> 42409 INFO (parallelCoreAdminExecutor-87-thread-1-process= ing-n:127.0.0.1:58398_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563589410= 29277 CREATE) [n:127.0.0.1:58398_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica3] o.a.s.c.CoreContainer Creating Solr= Core 'awhollynewcollection_0_shard1_replica3' using configuration from coll= ection awhollynewcollection_0 [junit4] 2> 42424 INFO (parallelCoreAdminExecutor-91-thread-1-process= ing-n:127.0.0.1:33161_solr 48e75d5b-d028-4bae-a0b4-7d4ada0f198c204563593405= 67608 CREATE) [n:127.0.0.1:33161_solr c:awhollynewcollection_0 s:shard1 x:= awhollynewcollection_0_shard1_replica2] o.a.s.s.IndexSchema [awhollynewcoll= ection_0_shard1_replica2] Schema name=3Dminimal [junit4] 2> 42440 INFO (parallelCoreAdminExecutor-91-thread-1-process= ing- [...truncated too long message...] onReplicationTest, TestSolr4Spatial, ZkNodePropsTest, TestJoin, TestFieldTy= peResource, TestRandomDVFaceting, SmileWriterTest, TestConfigSetsAPI, TestF= unctionQuery, DistributedSpellCheckComponentTest, ConcurrentDeleteAndCreate= CollectionTest, DefaultValueUpdateProcessorTest, TestSolrQueryParserDefault= OperatorResource, TestAnalyzeInfixSuggestions, TestSolrCloudWithSecureImper= sonation, TestFastWriter, TriLevelCompositeIdRoutingTest, HttpPartitionTest= , VersionInfoTest, TestSearcherReuse, TestSubQueryTransformer, TestFoldingM= ultitermQuery, RuleEngineTest, ActionThrottleTest, TestIBSimilarityFactory,= TestEmbeddedSolrServerConstructors, CurrencyFieldOpenExchangeTest, TestDow= nShardTolerantSearch, CdcrUpdateLogTest, TestCryptoKeys, SynonymTokenizerTe= st, ShardSplitTest, HighlighterMaxOffsetTest, WrapperMergePolicyFactoryTest= , DocValuesMissingTest, ZkStateReaderTest, TestNRTOpen, TestSolrDeletionPol= icy2, PathHierarchyTokenizerFactoryTest, SegmentsInfoRequestHandlerTest, Pa= rsingFieldUpdateProcessorsTest, TestRangeQuery, OverseerRolesTest, HdfsWrit= eToMultipleCollectionsTest, CdcrReplicationDistributedZkTest] [junit4] 2> NOTE: download the large Jenkins line-docs file by running= 'ant get-jenkins-line-docs' in the lucene directory. [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=3DCdcrReplicati= onDistributedZkTest -Dtests.seed=3D45D4F3CED8818740 -Dtests.multiplier=3D2 = -Dtests.nightly=3Dtrue -Dtests.slow=3Dtrue -Dtests.linedocsfile=3D/home/jen= kins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.= random.lines.txt -Dtests.locale=3Dvi -Dtests.timezone=3DIndian/Reunion -Dte= sts.asserts=3Dtrue -Dtests.file.encoding=3DUTF-8 [junit4] ERROR 0.00s J0 | CdcrReplicationDistributedZkTest (suite) <<< [junit4] > Throwable #1: java.lang.AssertionError: ObjectTracker foun= d 1 object(s) that were not released!!! [SolrZkClient] [junit4] > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTra= ckerException [junit4] > =09at org.apache.solr.common.util.ObjectReleaseTracker.tra= ck(ObjectReleaseTracker.java:43) [junit4] > =09at org.apache.solr.common.cloud.SolrZkClient.(Sol= rZkClient.java:184) [junit4] > =09at org.apache.solr.common.cloud.SolrZkClient.(Sol= rZkClient.java:116) [junit4] > =09at org.apache.solr.common.cloud.SolrZkClient.(Sol= rZkClient.java:106) [junit4] > =09at org.apache.solr.common.cloud.ZkStateReader.(Zk= StateReader.java:226) [junit4] > =09at org.apache.solr.client.solrj.impl.ZkClientClusterSta= teProvider.connect(ZkClientClusterStateProvider.java:121) [junit4] > =09at org.apache.solr.client.solrj.impl.CloudSolrClient.co= nnect(CloudSolrClient.java:610) [junit4] > =09at org.apache.solr.client.solrj.impl.CloudSolrClient.re= questWithRetryOnStaleState(CloudSolrClient.java:1048) [junit4] > =09at org.apache.solr.client.solrj.impl.CloudSolrClient.re= quest(CloudSolrClient.java:1037) [junit4] > =09at org.apache.solr.client.solrj.SolrClient.request(Solr= Client.java:1219) [junit4] > =09at org.apache.solr.handler.CdcrReplicatorManager.getChe= ckpoint(CdcrReplicatorManager.java:196) [junit4] > =09at org.apache.solr.handler.CdcrReplicatorManager.initLo= gReaders(CdcrReplicatorManager.java:159) [junit4] > =09at org.apache.solr.handler.CdcrReplicatorManager.stateU= pdate(CdcrReplicatorManager.java:134) [junit4] > =09at org.apache.solr.handler.CdcrStateManager.callback(Cd= crStateManager.java:36) [junit4] > =09at org.apache.solr.handler.CdcrLeaderStateManager.setAm= ILeader(CdcrLeaderStateManager.java:108) [junit4] > =09at org.apache.solr.handler.CdcrLeaderStateManager.check= IfIAmLeader(CdcrLeaderStateManager.java:95) [junit4] > =09at org.apache.solr.handler.CdcrLeaderStateManager.acces= s$400(CdcrLeaderStateManager.java:40) [junit4] > =09at org.apache.solr.handler.CdcrLeaderStateManager$Leade= rStateWatcher.process(CdcrLeaderStateManager.java:150) [junit4] > =09at org.apache.solr.common.cloud.SolrZkClient$3.lambda$p= rocess$0(SolrZkClient.java:268) [junit4] > =09at java.util.concurrent.Executors$RunnableAdapter.call(= Executors.java:511) [junit4] > =09at java.util.concurrent.FutureTask.run(FutureTask.java:= 266) [junit4] > =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThr= eadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) [junit4] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(Th= readPoolExecutor.java:1142) [junit4] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(T= hreadPoolExecutor.java:617) [junit4] > =09at java.lang.Thread.run(Thread.java:745) [junit4] > =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]= :0) [junit4] > =09at org.apache.solr.SolrTestCaseJ4.teardownTestCases(Sol= rTestCaseJ4.java:266) [junit4] > =09at java.lang.Thread.run(Thread.java:745)Throwable #2: c= om.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SU= ITE scope at org.apache.solr.cloud.CdcrReplicationDistributedZkTest:=20 [junit4] > 1) Thread[id=3D13299, name=3DzkCallback-2136-thread-5-p= rocessing-n:127.0.0.1:49190_m_sbe%2Fus-EventThread, state=3DWAITING, group= =3DTGRP-CdcrReplicationDistributedZkTest] [junit4] > at sun.misc.Unsafe.park(Native Method) [junit4] > at java.util.concurrent.locks.LockSupport.park(Loc= kSupport.java:175) [junit4] > at java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) [junit4] > at java.util.concurrent.LinkedBlockingQueue.take(L= inkedBlockingQueue.java:442) [junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run= (ClientCnxn.java:494) [junit4] > 2) Thread[id=3D13298, name=3DzkCallback-2136-thread-5-p= rocessing-n:127.0.0.1:49190_m_sbe%2Fus-SendThread(127.0.0.1:52678), state= =3DTIMED_WAITING, group=3DTGRP-CdcrReplicationDistributedZkTest] [junit4] > at java.lang.Thread.sleep(Native Method) [junit4] > at org.apache.zookeeper.client.StaticHostProvider.= next(StaticHostProvider.java:101) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.star= tConnect(ClientCnxn.java:940) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(= ClientCnxn.java:1003) [junit4] > =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]= :0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There = are still zombie threads that couldn't be terminated: [junit4] > 1) Thread[id=3D13298, name=3DzkCallback-2136-thread-5-p= rocessing-n:127.0.0.1:49190_m_sbe%2Fus-SendThread(127.0.0.1:52678), state= =3DTIMED_WAITING, group=3DTGRP-CdcrReplicationDistributedZkTest] [junit4] > at java.lang.Thread.sleep(Native Method) [junit4] > at org.apache.zookeeper.client.StaticHostProvider.= next(StaticHostProvider.java:101) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.star= tConnect(ClientCnxn.java:940) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(= ClientCnxn.java:1003) [junit4] > =09at __randomizedtesting.SeedInfo.seed([45D4F3CED8818740]= :0) [junit4] Completed [483/662 (3!)] on J0 in 1868.22s, 11 tests, 1 failure= , 2 errors <<< FAILURES! [...truncated 44827 lines...] -documentation-lint: [echo] checking for broken html... [jtidy] Checking for broken html (such as invalid tags)... [delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-S= olr-NightlyTests-6.x/checkout/lucene/build/jtidy_tmp [echo] Checking for broken links... [exec]=20 [exec] Crawl/parse... [exec]=20 [exec] Verify... [exec]=20 [exec] file:///build/docs/misc/org/apache/lucene/search/DocValuesStats= .DoubleDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.NumericDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.NumericDocValuesStats.html [exec]=20 [exec] file:///build/docs/misc/org/apache/lucene/search/DocValuesStats= .LongDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.NumericDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.NumericDocValuesStats.html [exec]=20 [exec] file:///build/docs/misc/org/apache/lucene/search/DocValuesStats= .SortedLongDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.SortedNumericDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.SortedNumericDocValuesStats.html [exec]=20 [exec] file:///build/docs/misc/org/apache/lucene/search/DocValuesStats= .SortedDoubleDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.SortedNumericDocValuesStats.html [exec] BROKEN LINK: file:///build/docs/core/org/apache/lucene/search= /DocValuesStats.SortedNumericDocValuesStats.html [exec]=20 [exec] Broken javadocs links were found! BUILD FAILED /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/b= uild.xml:773: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/b= uild.xml:101: The following error occurred while executing this line: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/l= ucene/build.xml:138: The following error occurred while executing this line= : /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/l= ucene/build.xml:151: The following error occurred while executing this line= : /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/l= ucene/common-build.xml:2478: exec returned: 1 Total time: 292 minutes 57 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any ------=_Part_1498_293239069.1482085549772 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org ------=_Part_1498_293239069.1482085549772--