Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A5758200CBB for ; Tue, 4 Jul 2017 13:35:00 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A20B7160C0B; Tue, 4 Jul 2017 11:35:00 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4894E160EC7 for ; Tue, 4 Jul 2017 13:34:58 +0200 (CEST) Received: (qmail 79813 invoked by uid 500); 4 Jul 2017 11:34:57 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 79803 invoked by uid 99); 4 Jul 2017 11:34:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jul 2017 11:34:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 4D5EB1A7B63 for ; Tue, 4 Jul 2017 11:34:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.3 X-Spam-Level: X-Spam-Status: No, score=-0.3 tagged_above=-999 required=6.31 tests=[KAM_BADIPHTTP=2, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id I-9JMgjjWdsw for ; Tue, 4 Jul 2017 11:34:41 +0000 (UTC) Received: from serv1.sd-datasolutions.de (serv1.sd-datasolutions.de [88.99.242.108]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 750FF5FAEA for ; Tue, 4 Jul 2017 11:34:40 +0000 (UTC) Received: from serv1 (ip6-localhost [IPv6:::1]) by serv1.sd-datasolutions.de (Postfix) with ESMTP id 8E304108062E for ; Tue, 4 Jul 2017 11:34:40 +0000 (UTC) Date: Tue, 4 Jul 2017 11:33:20 +0000 (UTC) From: Policeman Jenkins Server To: dev@lucene.apache.org Message-ID: <1541479117.297.1499168080583.JavaMail.jenkins@serv1.sd-datasolutions.de> In-Reply-To: <352810573.289.1499157819480.JavaMail.jenkins@serv1.sd-datasolutions.de> References: <352810573.289.1499157819480.JavaMail.jenkins@serv1.sd-datasolutions.de> Subject: [JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_131) - Build # 20055 - Still Unstable! MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_296_928765754.1499168080548" X-Jenkins-Job: Lucene-Solr-master-Linux X-Jenkins-Result: UNSTABLE archived-at: Tue, 04 Jul 2017 11:35:00 -0000 ------=_Part_296_928765754.1499168080548 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/20055/ Java: 64bit/jdk1.8.0_131 -XX:+UseCompressedOops -XX:+UseSerialGC 3 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothing= IsSafeWithPullReplicasTest Error Message: 3 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothi= ngIsSafeWithPullReplicasTest: 1) Thread[id=3D18002, name=3DConnection e= victor, state=3DTIMED_WAITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPul= lReplicasTest] at java.lang.Thread.sleep(Native Method) at = org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvict= or.java:66) at java.lang.Thread.run(Thread.java:748) 2) Thread[i= d=3D18003, name=3DTEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-se= ed#[372ABA285F670815]-SendThread(127.0.0.1:33003), state=3DTIMED_WAITING, g= roup=3DTGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.l= ang.Thread.sleep(Native Method) at org.apache.zookeeper.client.Stat= icHostProvider.next(StaticHostProvider.java:101) at org.apache.zook= eeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at or= g.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 3) Th= read[id=3D18004, name=3DTEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.t= est-seed#[372ABA285F670815]-EventThread, state=3DWAITING, group=3DTGRP-Chao= sMonkeyNothingIsSafeWithPullReplicasTest] at sun.misc.Unsafe.park(N= ative Method) at java.util.concurrent.locks.LockSupport.park(LockSu= pport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchro= nizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) a= t java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:44= 2) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.ja= va:501) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from S= UITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplica= sTest:=20 1) Thread[id=3D18002, name=3DConnection evictor, state=3DTIMED_WAITING, = group=3DTGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConn= ectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=3D18003, name=3DTEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]-SendThread(127.0.0.1:33003), state=3DTI= MED_WAITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostPr= ovider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCn= xn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1= 060) 3) Thread[id=3D18004, name=3DTEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]-EventThread, state=3DWAITING, group=3DT= GRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175= ) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionO= bject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueu= e.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:= 501) =09at __randomizedtesting.SeedInfo.seed([372ABA285F670815]:0) FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothing= IsSafeWithPullReplicasTest Error Message: There are still zombie threads that couldn't be terminated: 1) Thread[id= =3D18003, name=3DTEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-see= d#[372ABA285F670815]-SendThread(127.0.0.1:33003), state=3DTIMED_WAITING, gr= oup=3DTGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.la= ng.Thread.sleep(Native Method) at org.apache.zookeeper.client.Stati= cHostProvider.next(StaticHostProvider.java:101) at org.apache.zooke= eper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org= .apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie = threads that couldn't be terminated: 1) Thread[id=3D18003, name=3DTEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]-SendThread(127.0.0.1:33003), state=3DTI= MED_WAITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostPr= ovider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCn= xn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1= 060) =09at __randomizedtesting.SeedInfo.seed([372ABA285F670815]:0) FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest= .test Error Message: Timeout occured while waiting response from server at: http://127.0.0.1:434= 81 Stack Trace: org.apache.solr.client.solrj.SolrServerException: Timeout occured while wai= ting response from server at: http://127.0.0.1:43481 =09at __randomizedtesting.SeedInfo.seed([372ABA285F670815:BF7E85F2F19B65ED]= :0) =09at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSo= lrClient.java:637) =09at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClie= nt.java:252) =09at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClie= nt.java:241) =09at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSo= lrClient.java:483) =09at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolr= Client.java:413) =09at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSo= lrClient.java:1121) =09at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnS= taleState(CloudSolrClient.java:862) =09at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrCl= ient.java:793) =09at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) =09at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(= AbstractFullDistribZkTestBase.java:1667) =09at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(= AbstractFullDistribZkTestBase.java:1694) =09at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.te= st(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:297) =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:62) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1713) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Random= izedRunner.java:907) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(Random= izedRunner.java:943) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(Rando= mizedRunner.java:957) =09at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$Shards= FixedStatement.callStatement(BaseDistributedSearchTestCase.java:985) =09at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$Shards= Statement.evaluate(BaseDistributedSearchTestCase.java:960) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRu= leSetupTeardownChained.java:49) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleT= hreadAndTestName.java:48) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:368) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTa= sk(ThreadLeakControl.java:817) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(Threa= dLeakControl.java:468) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran= domizedRunner.java:916) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Random= izedRunner.java:802) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random= izedRunner.java:852) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:863) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:368) =09at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketTimeoutException: Read timed out =09at java.net.SocketInputStream.socketRead0(Native Method) =09at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) =09at java.net.SocketInputStream.read(SocketInputStream.java:171) =09at java.net.SocketInputStream.read(SocketInputStream.java:141) =09at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInpu= tBufferImpl.java:139) =09at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInpu= tBufferImpl.java:155) =09at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputB= ufferImpl.java:284) =09at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(Default= HttpResponseParser.java:140) =09at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(Default= HttpResponseParser.java:57) =09at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessagePa= rser.java:261) =09at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHead= er(DefaultBHttpClientConnection.java:165) =09at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy= .java:167) =09at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRe= questExecutor.java:272) =09at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecu= tor.java:124) =09at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.= java:271) =09at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java= :184) =09at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) =09at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java= :110) =09at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttp= Client.java:184) =09at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttp= Client.java:82) =09at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttp= Client.java:55) =09at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSo= lrClient.java:523) =09... 52 more Build Log: [...truncated 12063 lines...] [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullRe= plicasTest [junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-mast= er-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsS= afeWithPullReplicasTest_372ABA285F670815-001/init-core-data-001 [junit4] 2> 1122514 WARN (SUITE-ChaosMonkeyNothingIsSafeWithPullRepli= casTest-seed#[372ABA285F670815]-worker) [ ] o.a.s.SolrTestCaseJ4 startTr= ackingSearchers: numOpens=3D26 numCloses=3D26 [junit4] 2> 1122514 INFO (SUITE-ChaosMonkeyNothingIsSafeWithPullRepli= casTest-seed#[372ABA285F670815]-worker) [ ] o.a.s.SolrTestCaseJ4 Using T= rieFields (NUMERIC_POINTS_SYSPROP=3Dfalse) w/NUMERIC_DOCVALUES_SYSPROP=3Dfa= lse [junit4] 2> 1122515 INFO (SUITE-ChaosMonkeyNothingIsSafeWithPullRepli= casTest-seed#[372ABA285F670815]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomi= zed ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4= $SuppressSSL(bugUrl=3Dhttps://issues.apache.org/jira/browse/SOLR-5776) [junit4] 2> 1122515 INFO (SUITE-ChaosMonkeyNothingIsSafeWithPullRepli= casTest-seed#[372ABA285F670815]-worker) [ ] o.a.s.BaseDistributedSearchT= estCase Setting hostContext system property: / [junit4] 2> 1122516 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ChaosMonkeyNothingIsSaf= eWithPullReplicasTest Starting ChaosMonkey test with 2 shards and 4 nodes [junit4] 2> 1122517 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ZkTestServer STARTING Z= K TEST SERVER [junit4] 2> 1122517 INFO (Thread-6717) [ ] o.a.s.c.ZkTestServer cl= ient port:0.0.0.0/0.0.0.0:0 [junit4] 2> 1122518 INFO (Thread-6717) [ ] o.a.s.c.ZkTestServer St= arting server [junit4] 2> 1122519 ERROR (Thread-6717) [ ] o.a.z.s.ZooKeeperServer= ZKShutdownHandler is not registered, so ZooKeeper server won't take any ac= tion on ERROR or SHUTDOWN server state changes [junit4] 2> 1122618 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ZkTestServer start zk s= erver on port:33003 [junit4] 2> 1122627 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml [junit4] 2> 1122627 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/schema15.xml to /configs/conf1/schema.xml [junit4] 2> 1122628 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/c= onf1/solrconfig.snippet.randomindexconfig.xml [junit4] 2> 1122628 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4] 2> 1122629 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4] 2> 1122629 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4] 2> 1122629 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml [junit4] 2> 1122630 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchan= ge-rates.json [junit4] 2> 1122630 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-= ISOLatin1Accent.txt [junit4] 2> 1122630 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4] 2> 1122630 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractZkTestCase put = /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test-files/s= olr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4] 2> 1122631 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractFullDistribZkTe= stBase Will use TLOG replicas unless explicitly asked otherwise [junit4] 2> 1122698 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server jetty-9.3.14.v20= 161028 [junit4] 2> 1122699 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.h.ContextHandler Starte= d o.e.j.s.ServletContextHandler@14ccbb3{/,null,AVAILABLE} [junit4] 2> 1122700 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.AbstractConnector Start= ed ServerConnector@89e9d31{HTTP/1.1,[http/1.1]}{127.0.0.1:43977} [junit4] 2> 1122700 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server Started @1124063= ms [junit4] 2> 1122700 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.s.e.JettySolrRunner Jet= ty properties: {solr.data.dir=3D/home/jenkins/workspace/Lucene-Solr-master-= Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafe= WithPullReplicasTest_372ABA285F670815-001/tempDir-001/control/data, replica= Type=3DNRT, hostContext=3D/, hostPort=3D43977, coreRootDirectory=3D/home/je= nkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/= solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_372ABA285F670815-00= 1/control-001/cores} [junit4] 2> 1122701 ERROR (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.StartupLoggingUtils Mis= sing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1122701 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter ___= _ Welcome to Apache Solr=E2=84=A2 version 8.0.0 [junit4] 2> 1122701 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter / __= | ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1122701 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter \__ = \/ _ \ | '_| Install dir: null, Default config dir: null [junit4] 2> 1122701 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter |___= /\___/_|_| Start time: 2017-07-04T11:05:22.374Z [junit4] 2> 1122702 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter Load= ing solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1122702 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig Loading c= ontainer configuration from /home/jenkins/workspace/Lucene-Solr-master-Linu= x/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWith= PullReplicasTest_372ABA285F670815-001/control-001/solr.xml [junit4] 2> 1122705 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig MBean ser= ver found: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd, but no JMX repo= rters were configured - adding default JMX reporter. [junit4] 2> 1122706 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ZkContainer Zookeeper c= lient=3D127.0.0.1:33003/solr [junit4] 2> 1122728 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.c.Overs= eer Overseer (id=3Dnull) closing [junit4] 2> 1122728 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.c.Overs= eerElectionContext I am going to be the leader 127.0.0.1:43977_ [junit4] 2> 1122728 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.c.Overs= eer Overseer (id=3D98249364091109380-127.0.0.1:43977_-n_0000000000) startin= g [junit4] 2> 1122730 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.c.ZkCon= troller Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43977_ [junit4] 2> 1122730 INFO (zkCallback-2389-thread-1-processing-n:127.0= .0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1122813 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled = at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1122816 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at= server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1122816 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enable= d at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1122817 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43977_ ] o.a.s.c.CoreP= ropertiesLocator Found 0 core definitions underneath /home/jenkins/workspac= e/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.Cha= osMonkeyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/control-001/= cores [junit4] 2> 1122830 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.c.ZkStateReader Updated= live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1122831 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.s.i.ZkClientClusterStat= eProvider Cluster at 127.0.0.1:33003/solr ready [junit4] 2> 1122832 INFO (qtp1478039675-17971) [n:127.0.0.1:43977_ = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with para= ms replicationFactor=3D1&collection.configName=3Dconf1&name=3Dcontrol_colle= ction&nrtReplicas=3D1&action=3DCREATE&numShards=3D1&createNodeSet=3D127.0.0= .1:43977_&wt=3Djavabin&version=3D2 and sendToOCPQueue=3Dtrue [junit4] 2> 1122833 INFO (OverseerThreadFactory-6610-thread-1-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.CreateCollectionCm= d Create collection control_collection [junit4] 2> 1122936 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ = ] o.a.s.h.a.CoreAdminOperation core create command qt=3D/admin/cores&colle= ction.configName=3Dconf1&newCollection=3Dtrue&name=3Dcontrol_collection_sha= rd1_replica_n1&action=3DCREATE&numShards=3D1&collection=3Dcontrol_collectio= n&shard=3Dshard1&wt=3Djavabin&version=3D2&replicaType=3DNRT [junit4] 2> 1122937 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ = ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 t= ransient cores [junit4] 2> 1123040 INFO (zkCallback-2389-thread-1-processing-n:127.0= .0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/control_collection/state.json] for collection [control_collection]= has occurred - updating... (live nodes size: [1]) [junit4] 2> 1123954 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.SolrConfig Using Lucene MatchVersion: 8.0.0 [junit4] 2> 1123969 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= s.IndexSchema [control_collection_shard1_replica_n1] Schema name=3Dtest [junit4] 2> 1124026 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= s.IndexSchema Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1124034 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' us= ing configuration from collection control_collection, trusted=3Dtrue [junit4] 2> 1124034 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1= .replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') en= abled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1124034 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.SolrCore solr.RecoveryStrategy.Builder [junit4] 2> 1124034 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore a= t [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/te= st/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_372ABA28= 5F670815-001/control-001/cores/control_collection_shard1_replica_n1], dataD= ir=3D[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core= /test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_372AB= A285F670815-001/control-001/cores/control_collection_shard1_replica_n1/data= /] [junit4] 2> 1124036 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.inde= x.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3D26, maxMergeAtOnc= eExplicit=3D50, maxMergedSegmentMB=3D49.140625, floorSegmentMB=3D0.75488281= 25, forceMergeDeletesPctAllowed=3D13.188791359168574, segmentsPerTier=3D23.= 0, maxCFSSegmentSizeMB=3D8.796093022207999E12, noCFSRatio=3D0.0 [junit4] 2> 1124054 WARN (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.RequestHandlers INVALID paramSet a in requestHandler {type =3D requestHan= dler,name =3D /dump,class =3D DumpRequestHandler,attributes =3D {initParams= =3Da, name=3D/dump, class=3DDumpRequestHandler},args =3D {defaults=3D{a=3DA= ,b=3DB}}} [junit4] 2> 1124092 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.Upda= teLog [junit4] 2> 1124092 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= u.UpdateLog Initializing UpdateLog: dataDir=3D defaultSyncLevel=3DFLUSH num= RecordsToKeep=3D100 maxNumLogsToKeep=3D10 numVersionBuckets=3D65536 [junit4] 2> 1124093 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;=20 [junit4] 2> 1124093 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 1124094 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.inde= x.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=3D1000, mergeFactor= =3D24, maxMergeSize=3D9223372036854775807, maxMergeSizeForForcedMerge=3D922= 3372036854775807, calibrateSizeByDeletes=3Dtrue, maxMergeDocs=3D2147483647,= maxCFSSegmentSizeMB=3D8.796093022207999E12, noCFSRatio=3D0.345068259576717= 05] [junit4] 2> 1124094 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= s.SolrIndexSearcher Opening [Searcher@28695146[control_collection_shard1_re= plica_n1] main] [junit4] 2> 1124095 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /con= figs/conf1 [junit4] 2> 1124095 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKe= eperStorageIO:path=3D/configs/conf1 [junit4] 2> 1124096 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 1124096 INFO (searcherExecutor-6613-thread-1-processing-n= :127.0.0.1:43977_ x:control_collection_shard1_replica_n1 s:shard1 c:control= _collection) [n:127.0.0.1:43977_ c:control_collection s:shard1 x:control_c= ollection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_re= plica_n1] Registered new searcher Searcher@28695146[control_collection_shar= d1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 1124096 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= u.UpdateLog Could not find max version in index or recent updates, using ne= w clock 1571989827112402944 [junit4] 2> 1124100 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 1124100 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 1124100 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.SyncStrategy Sync replicas to http://127.0.0.1:43977/control_collection_s= hard1_replica_n1/ [junit4] 2> 1124100 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.SyncStrategy Sync Success - now sync replicas to me [junit4] 2> 1124100 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.SyncStrategy http://127.0.0.1:43977/control_collection_shard1_replica_n1/= has no replicas [junit4] 2> 1124100 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.ShardLeaderElectionContext Found all replicas participating in election, = clear LIR [junit4] 2> 1124101 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43977/co= ntrol_collection_shard1_replica_n1/ shard1 [junit4] 2> 1124202 INFO (zkCallback-2389-thread-1-processing-n:127.0= .0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/control_collection/state.json] for collection [control_collection]= has occurred - updating... (live nodes size: [1]) [junit4] 2> 1124251 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= c.ZkController I am the leader, no recovery necessary [junit4] 2> 1124253 INFO (qtp1478039675-17973) [n:127.0.0.1:43977_ c:= control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.= s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{qt=3D/ad= min/cores&collection.configName=3Dconf1&newCollection=3Dtrue&name=3Dcontrol= _collection_shard1_replica_n1&action=3DCREATE&numShards=3D1&collection=3Dco= ntrol_collection&shard=3Dshard1&wt=3Djavabin&version=3D2&replicaType=3DNRT}= status=3D0 QTime=3D1317 [junit4] 2> 1124255 INFO (qtp1478039675-17971) [n:127.0.0.1:43977_ = ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at= most 30 seconds. Check all shard replicas [junit4] 2> 1124355 INFO (zkCallback-2389-thread-1-processing-n:127.0= .0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/control_collection/state.json] for collection [control_collection]= has occurred - updating... (live nodes size: [1]) [junit4] 2> 1124835 INFO (OverseerCollectionConfigSetProcessor-982493= 64091109380-127.0.0.1:43977_-n_0000000000) [n:127.0.0.1:43977_ ] o.a.s.c= .OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-00= 00000000 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1125256 INFO (qtp1478039675-17971) [n:127.0.0.1:43977_ = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections par= ams=3D{replicationFactor=3D1&collection.configName=3Dconf1&name=3Dcontrol_c= ollection&nrtReplicas=3D1&action=3DCREATE&numShards=3D1&createNodeSet=3D127= .0.0.1:43977_&wt=3Djavabin&version=3D2} status=3D0 QTime=3D2424 [junit4] 2> 1125261 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.c.ZkStateReader Updated= live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1125262 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.s.i.ZkClientClusterStat= eProvider Cluster at 127.0.0.1:33003/solr ready [junit4] 2> 1125262 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ChaosMonkey monkey: ini= t - expire sessions:false cause connection loss:false [junit4] 2> 1125263 INFO (qtp1478039675-17971) [n:127.0.0.1:43977_ = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with para= ms replicationFactor=3D1&collection.configName=3Dconf1&name=3Dcollection1&n= rtReplicas=3D1&action=3DCREATE&numShards=3D2&createNodeSet=3D&stateFormat= =3D2&wt=3Djavabin&version=3D2 and sendToOCPQueue=3Dtrue [junit4] 2> 1125264 INFO (OverseerThreadFactory-6610-thread-2-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.CreateCollectionCm= d Create collection collection1 [junit4] 2> 1125265 WARN (OverseerThreadFactory-6610-thread-2-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.CreateCollectionCm= d It is unusual to create a collection (collection1) without cores. [junit4] 2> 1125469 INFO (qtp1478039675-17971) [n:127.0.0.1:43977_ = ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at= most 30 seconds. Check all shard replicas [junit4] 2> 1125470 INFO (qtp1478039675-17971) [n:127.0.0.1:43977_ = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections par= ams=3D{replicationFactor=3D1&collection.configName=3Dconf1&name=3Dcollectio= n1&nrtReplicas=3D1&action=3DCREATE&numShards=3D2&createNodeSet=3D&stateForm= at=3D2&wt=3Djavabin&version=3D2} status=3D0 QTime=3D207 [junit4] 2> 1125540 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractFullDistribZkTe= stBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-mast= er-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsS= afeWithPullReplicasTest_372ABA285F670815-001/shard-1-001 of type TLOG [junit4] 2> 1125540 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server jetty-9.3.14.v20= 161028 [junit4] 2> 1125541 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.h.ContextHandler Starte= d o.e.j.s.ServletContextHandler@fc04789{/,null,AVAILABLE} [junit4] 2> 1125542 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.AbstractConnector Start= ed ServerConnector@16ad01e4{HTTP/1.1,[http/1.1]}{127.0.0.1:38085} [junit4] 2> 1125542 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server Started @1126904= ms [junit4] 2> 1125542 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.s.e.JettySolrRunner Jet= ty properties: {solr.data.dir=3D/home/jenkins/workspace/Lucene-Solr-master-= Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafe= WithPullReplicasTest_372ABA285F670815-001/tempDir-001/jetty1, replicaType= =3DTLOG, solrconfig=3Dsolrconfig.xml, hostContext=3D/, hostPort=3D38085, co= reRootDirectory=3D/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/bui= ld/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Luce= ne-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonk= eyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/shard-1-001/cores} [junit4] 2> 1125542 ERROR (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.StartupLoggingUtils Mis= sing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1125542 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter ___= _ Welcome to Apache Solr=E2=84=A2 version 8.0.0 [junit4] 2> 1125542 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter / __= | ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1125542 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter \__ = \/ _ \ | '_| Install dir: null, Default config dir: null [junit4] 2> 1125542 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter |___= /\___/_|_| Start time: 2017-07-04T11:05:25.215Z [junit4] 2> 1125544 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter Load= ing solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1125544 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig Loading c= ontainer configuration from /home/jenkins/workspace/Lucene-Solr-master-Linu= x/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWith= PullReplicasTest_372ABA285F670815-001/shard-1-001/solr.xml [junit4] 2> 1125546 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig MBean ser= ver found: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd, but no JMX repo= rters were configured - adding default JMX reporter. [junit4] 2> 1125549 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ZkContainer Zookeeper c= lient=3D127.0.0.1:33003/solr [junit4] 2> 1125555 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkS= tateReader Updated live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 1125555 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:38085_ ] o.a.s.c.Overs= eer Overseer (id=3Dnull) closing [junit4] 2> 1125556 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:38085_ ] o.a.s.c.ZkCon= troller Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38085_ [junit4] 2> 1125556 INFO (zkCallback-2396-thread-1) [ ] o.a.s.c.c.= ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1125556 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1125556 INFO (zkCallback-2389-thread-1-processing-n:127.0= .0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (1) -> (2) [junit4] 2> 1125610 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:38085_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled = at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1125613 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:38085_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at= server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1125613 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:38085_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enable= d at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1125614 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:38085_ ] o.a.s.c.CoreP= ropertiesLocator Found 0 core definitions underneath /home/jenkins/workspac= e/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../.= ./../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-cor= e/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_372A= BA285F670815-001/shard-1-001/cores [junit4] 2> 1125625 INFO (qtp2071608789-18012) [n:127.0.0.1:38085_ = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with = params node=3D127.0.0.1:38085_&action=3DADDREPLICA&collection=3Dcollection1= &shard=3Dshard2&type=3DTLOG&wt=3Djavabin&version=3D2 and sendToOCPQueue=3Dt= rue [junit4] 2> 1125626 INFO (OverseerThreadFactory-6610-thread-3-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.AddReplicaCmd addR= eplica() : { [junit4] 2> "collection":"collection1", [junit4] 2> "node":"127.0.0.1:38085_", [junit4] 2> "shard":"shard2", [junit4] 2> "type":"TLOG", [junit4] 2> "operation":"addreplica"} [junit4] 2> 1125626 INFO (OverseerCollectionConfigSetProcessor-982493= 64091109380-127.0.0.1:43977_-n_0000000000) [n:127.0.0.1:43977_ ] o.a.s.c= .OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-00= 00000002 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1125626 INFO (OverseerThreadFactory-6610-thread-3-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.AddReplicaCmd Node= Identified 127.0.0.1:38085_ for creating new replica [junit4] 2> 1125627 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ = ] o.a.s.h.a.CoreAdminOperation core create command qt=3D/admin/cores&colle= ction.configName=3Dconf1&name=3Dcollection1_shard2_replica_t0&action=3DCREA= TE&collection=3Dcollection1&shard=3Dshard2&wt=3Djavabin&version=3D2&replica= Type=3DTLOG [junit4] 2> 1125627 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ = ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 t= ransient cores [junit4] 2> 1125729 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [2]) [junit4] 2> 1126643 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.SolrConfig U= sing Lucene MatchVersion: 8.0.0 [junit4] 2> 1126683 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.s.IndexSchema = [collection1_shard2_replica_t0] Schema name=3Dtest [junit4] 2> 1126740 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.s.IndexSchema = Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1126745 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.CoreContaine= r Creating SolrCore 'collection1_shard2_replica_t0' using configuration fro= m collection collection1, trusted=3Dtrue [junit4] 2> 1126745 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.m.r.SolrJmxRep= orter JMX monitoring for 'solr.core.collection1.shard2.replica_t0' (registr= y 'solr.core.collection1.shard2.replica_t0') enabled at server: com.sun.jmx= .mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1126745 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.SolrCore sol= r.RecoveryStrategy.Builder [junit4] 2> 1126745 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.SolrCore [[c= ollection1_shard2_replica_t0] ] Opening new SolrCore at [/home/jenkins/work= space/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud= .ChaosMonkeyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/shard-1-= 001/cores/collection1_shard2_replica_t0], dataDir=3D[/home/jenkins/workspac= e/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../.= ./../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-cor= e/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_372A= BA285F670815-001/shard-1-001/cores/collection1_shard2_replica_t0/data/] [junit4] 2> 1126747 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.u.RandomMergeP= olicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergeP= olicy: [TieredMergePolicy: maxMergeAtOnce=3D26, maxMergeAtOnceExplicit=3D50= , maxMergedSegmentMB=3D49.140625, floorSegmentMB=3D0.7548828125, forceMerge= DeletesPctAllowed=3D13.188791359168574, segmentsPerTier=3D23.0, maxCFSSegme= ntSizeMB=3D8.796093022207999E12, noCFSRatio=3D0.0 [junit4] 2> 1126762 WARN (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.RequestHandl= ers INVALID paramSet a in requestHandler {type =3D requestHandler,name =3D = /dump,class =3D DumpRequestHandler,attributes =3D {initParams=3Da, name=3D/= dump, class=3DDumpRequestHandler},args =3D {defaults=3D{a=3DA,b=3DB}}} [junit4] 2> 1126833 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.u.UpdateHandle= r Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1126833 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.u.UpdateLog In= itializing UpdateLog: dataDir=3D defaultSyncLevel=3DFLUSH numRecordsToKeep= =3D100 maxNumLogsToKeep=3D10 numVersionBuckets=3D65536 [junit4] 2> 1126834 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.u.CommitTracke= r Hard AutoCommit: if uncommited for 15000ms;=20 [junit4] 2> 1126834 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.u.CommitTracke= r Soft AutoCommit: disabled [junit4] 2> 1126835 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.u.RandomMergeP= olicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergeP= olicy: [LogDocMergePolicy: minMergeSize=3D1000, mergeFactor=3D24, maxMergeS= ize=3D9223372036854775807, maxMergeSizeForForcedMerge=3D9223372036854775807= , calibrateSizeByDeletes=3Dtrue, maxMergeDocs=3D2147483647, maxCFSSegmentSi= zeMB=3D8.796093022207999E12, noCFSRatio=3D0.34506825957671705] [junit4] 2> 1126836 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.s.SolrIndexSea= rcher Opening [Searcher@7e24cea8[collection1_shard2_replica_t0] main] [junit4] 2> 1126836 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.r.ManagedResou= rceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1126837 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.r.ManagedResou= rceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:= path=3D/configs/conf1 [junit4] 2> 1126837 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.h.ReplicationH= andler Commits will be reserved for 10000 [junit4] 2> 1126838 INFO (searcherExecutor-6624-thread-1-processing-n= :127.0.0.1:38085_ x:collection1_shard2_replica_t0 s:shard2 c:collection1) [= n:127.0.0.1:38085_ c:collection1 s:shard2 x:collection1_shard2_replica_t0]= o.a.s.c.SolrCore [collection1_shard2_replica_t0] Registered new searcher S= earcher@7e24cea8[collection1_shard2_replica_t0] main{ExitableDirectoryReade= r(UninvertingDirectoryReader())} [junit4] 2> 1126838 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.u.UpdateLog Co= uld not find max version in index or recent updates, using new clock 157198= 9829987598336 [junit4] 2> 1126841 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderE= lectionContext Enough replicas found to continue. [junit4] 2> 1126841 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderE= lectionContext I may be the new leader - try and sync [junit4] 2> 1126841 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy= Sync replicas to http://127.0.0.1:38085/collection1_shard2_replica_t0/ [junit4] 2> 1126842 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy= Sync Success - now sync replicas to me [junit4] 2> 1126842 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.SyncStrategy= http://127.0.0.1:38085/collection1_shard2_replica_t0/ has no replicas [junit4] 2> 1126842 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderE= lectionContext Found all replicas participating in election, clear LIR [junit4] 2> 1126842 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.ZkController= collection1_shard2_replica_t0 stopping background replication from leader [junit4] 2> 1126843 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.ShardLeaderE= lectionContext I am the new leader: http://127.0.0.1:38085/collection1_shar= d2_replica_t0/ shard2 [junit4] 2> 1126944 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [2]) [junit4] 2> 1126993 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.c.ZkController= I am the leader, no recovery necessary [junit4] 2> 1126995 INFO (qtp2071608789-18014) [n:127.0.0.1:38085_ c:= collection1 s:shard2 x:collection1_shard2_replica_t0] o.a.s.s.HttpSolrCall= [admin] webapp=3Dnull path=3D/admin/cores params=3D{qt=3D/admin/cores&coll= ection.configName=3Dconf1&name=3Dcollection1_shard2_replica_t0&action=3DCRE= ATE&collection=3Dcollection1&shard=3Dshard2&wt=3Djavabin&version=3D2&replic= aType=3DTLOG} status=3D0 QTime=3D1368 [junit4] 2> 1126996 INFO (qtp2071608789-18012) [n:127.0.0.1:38085_ = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections par= ams=3D{node=3D127.0.0.1:38085_&action=3DADDREPLICA&collection=3Dcollection1= &shard=3Dshard2&type=3DTLOG&wt=3Djavabin&version=3D2} status=3D0 QTime=3D13= 71 [junit4] 2> 1127077 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractFullDistribZkTe= stBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-mast= er-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsS= afeWithPullReplicasTest_372ABA285F670815-001/shard-2-001 of type TLOG [junit4] 2> 1127077 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server jetty-9.3.14.v20= 161028 [junit4] 2> 1127078 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.h.ContextHandler Starte= d o.e.j.s.ServletContextHandler@ca1a573{/,null,AVAILABLE} [junit4] 2> 1127078 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.AbstractConnector Start= ed ServerConnector@11cd6019{HTTP/1.1,[http/1.1]}{127.0.0.1:43481} [junit4] 2> 1127078 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server Started @1128441= ms [junit4] 2> 1127078 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.s.e.JettySolrRunner Jet= ty properties: {solr.data.dir=3D/home/jenkins/workspace/Lucene-Solr-master-= Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafe= WithPullReplicasTest_372ABA285F670815-001/tempDir-001/jetty2, replicaType= =3DTLOG, solrconfig=3Dsolrconfig.xml, hostContext=3D/, hostPort=3D43481, co= reRootDirectory=3D/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/bui= ld/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Luce= ne-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonk= eyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/shard-2-001/cores} [junit4] 2> 1127079 ERROR (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.StartupLoggingUtils Mis= sing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1127079 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter ___= _ Welcome to Apache Solr=E2=84=A2 version 8.0.0 [junit4] 2> 1127079 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter / __= | ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1127079 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter \__ = \/ _ \ | '_| Install dir: null, Default config dir: null [junit4] 2> 1127079 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter |___= /\___/_|_| Start time: 2017-07-04T11:05:26.752Z [junit4] 2> 1127080 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter Load= ing solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1127080 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig Loading c= ontainer configuration from /home/jenkins/workspace/Lucene-Solr-master-Linu= x/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWith= PullReplicasTest_372ABA285F670815-001/shard-2-001/solr.xml [junit4] 2> 1127082 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig MBean ser= ver found: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd, but no JMX repo= rters were configured - adding default JMX reporter. [junit4] 2> 1127084 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ZkContainer Zookeeper c= lient=3D127.0.0.1:33003/solr [junit4] 2> 1127085 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ = ] o.a.z.s.NIOServerCnxn caught end of stream exception [junit4] 2> EndOfStreamException: Unable to read additional data from = client sessionid 0x15d0d46ae0f000b, likely client has closed socket [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOSe= rverCnxn.java:239) [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxnFactory.run= (NIOServerCnxnFactory.java:203) [junit4] 2> =09at java.lang.Thread.run(Thread.java:748) [junit4] 2> 1127087 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43481_ ] o.a.s.c.c.ZkS= tateReader Updated live nodes from ZooKeeper... (0) -> (2) [junit4] 2> 1127088 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43481_ ] o.a.s.c.Overs= eer Overseer (id=3Dnull) closing [junit4] 2> 1127089 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43481_ ] o.a.s.c.ZkCon= troller Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43481_ [junit4] 2> 1127089 INFO (zkCallback-2389-thread-2-processing-n:127.0= .0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1127089 INFO (zkCallback-2396-thread-1) [ ] o.a.s.c.c.= ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1127089 INFO (zkCallback-2407-thread-1-processing-n:127.0= .0.1:43481_) [n:127.0.0.1:43481_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1127089 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (2) -> (3) [junit4] 2> 1127132 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43481_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled = at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1127135 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43481_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at= server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1127135 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43481_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enable= d at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1127135 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:43481_ ] o.a.s.c.CoreP= ropertiesLocator Found 0 core definitions underneath /home/jenkins/workspac= e/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../.= ./../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-cor= e/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_372A= BA285F670815-001/shard-2-001/cores [junit4] 2> 1127149 INFO (qtp2071608789-18013) [n:127.0.0.1:38085_ = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with = params node=3D127.0.0.1:43481_&action=3DADDREPLICA&collection=3Dcollection1= &shard=3Dshard1&type=3DTLOG&wt=3Djavabin&version=3D2 and sendToOCPQueue=3Dt= rue [junit4] 2> 1127150 INFO (OverseerThreadFactory-6610-thread-4-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.AddReplicaCmd addR= eplica() : { [junit4] 2> "collection":"collection1", [junit4] 2> "node":"127.0.0.1:43481_", [junit4] 2> "shard":"shard1", [junit4] 2> "type":"TLOG", [junit4] 2> "operation":"addreplica"} [junit4] 2> 1127150 INFO (OverseerCollectionConfigSetProcessor-982493= 64091109380-127.0.0.1:43977_-n_0000000000) [n:127.0.0.1:43977_ ] o.a.s.c= .OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-00= 00000004 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1127150 INFO (OverseerThreadFactory-6610-thread-4-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.AddReplicaCmd Node= Identified 127.0.0.1:43481_ for creating new replica [junit4] 2> 1127151 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ = ] o.a.s.h.a.CoreAdminOperation core create command qt=3D/admin/cores&colle= ction.configName=3Dconf1&name=3Dcollection1_shard1_replica_t0&action=3DCREA= TE&collection=3Dcollection1&shard=3Dshard1&wt=3Djavabin&version=3D2&replica= Type=3DTLOG [junit4] 2> 1127151 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ = ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 t= ransient cores [junit4] 2> 1127253 INFO (zkCallback-2407-thread-1-processing-n:127.0= .0.1:43481_) [n:127.0.0.1:43481_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [3]) [junit4] 2> 1127253 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [3]) [junit4] 2> 1128166 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.SolrConfig U= sing Lucene MatchVersion: 8.0.0 [junit4] 2> 1128182 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.s.IndexSchema = [collection1_shard1_replica_t0] Schema name=3Dtest [junit4] 2> 1128246 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.s.IndexSchema = Loaded schema test/1.6 with uniqueid field id [junit4] 2> 1128251 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.CoreContaine= r Creating SolrCore 'collection1_shard1_replica_t0' using configuration fro= m collection collection1, trusted=3Dtrue [junit4] 2> 1128252 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.m.r.SolrJmxRep= orter JMX monitoring for 'solr.core.collection1.shard1.replica_t0' (registr= y 'solr.core.collection1.shard1.replica_t0') enabled at server: com.sun.jmx= .mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1128252 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.SolrCore sol= r.RecoveryStrategy.Builder [junit4] 2> 1128252 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.SolrCore [[c= ollection1_shard1_replica_t0] ] Opening new SolrCore at [/home/jenkins/work= space/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud= .ChaosMonkeyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/shard-2-= 001/cores/collection1_shard1_replica_t0], dataDir=3D[/home/jenkins/workspac= e/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/../../../../../../.= ./../../home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-cor= e/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest_372A= BA285F670815-001/shard-2-001/cores/collection1_shard1_replica_t0/data/] [junit4] 2> 1128254 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.u.RandomMergeP= olicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergeP= olicy: [TieredMergePolicy: maxMergeAtOnce=3D26, maxMergeAtOnceExplicit=3D50= , maxMergedSegmentMB=3D49.140625, floorSegmentMB=3D0.7548828125, forceMerge= DeletesPctAllowed=3D13.188791359168574, segmentsPerTier=3D23.0, maxCFSSegme= ntSizeMB=3D8.796093022207999E12, noCFSRatio=3D0.0 [junit4] 2> 1128271 WARN (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.RequestHandl= ers INVALID paramSet a in requestHandler {type =3D requestHandler,name =3D = /dump,class =3D DumpRequestHandler,attributes =3D {initParams=3Da, name=3D/= dump, class=3DDumpRequestHandler},args =3D {defaults=3D{a=3DA,b=3DB}}} [junit4] 2> 1128295 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.u.UpdateHandle= r Using UpdateLog implementation: org.apache.solr.update.UpdateLog [junit4] 2> 1128296 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.u.UpdateLog In= itializing UpdateLog: dataDir=3D defaultSyncLevel=3DFLUSH numRecordsToKeep= =3D100 maxNumLogsToKeep=3D10 numVersionBuckets=3D65536 [junit4] 2> 1128296 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.u.CommitTracke= r Hard AutoCommit: if uncommited for 15000ms;=20 [junit4] 2> 1128296 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.u.CommitTracke= r Soft AutoCommit: disabled [junit4] 2> 1128297 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.u.RandomMergeP= olicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergeP= olicy: [LogDocMergePolicy: minMergeSize=3D1000, mergeFactor=3D24, maxMergeS= ize=3D9223372036854775807, maxMergeSizeForForcedMerge=3D9223372036854775807= , calibrateSizeByDeletes=3Dtrue, maxMergeDocs=3D2147483647, maxCFSSegmentSi= zeMB=3D8.796093022207999E12, noCFSRatio=3D0.34506825957671705] [junit4] 2> 1128297 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.s.SolrIndexSea= rcher Opening [Searcher@6081f6d7[collection1_shard1_replica_t0] main] [junit4] 2> 1128298 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.r.ManagedResou= rceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1128298 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.r.ManagedResou= rceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:= path=3D/configs/conf1 [junit4] 2> 1128298 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.h.ReplicationH= andler Commits will be reserved for 10000 [junit4] 2> 1128299 INFO (searcherExecutor-6635-thread-1-processing-n= :127.0.0.1:43481_ x:collection1_shard1_replica_t0 s:shard1 c:collection1) [= n:127.0.0.1:43481_ c:collection1 s:shard1 x:collection1_shard1_replica_t0]= o.a.s.c.SolrCore [collection1_shard1_replica_t0] Registered new searcher S= earcher@6081f6d7[collection1_shard1_replica_t0] main{ExitableDirectoryReade= r(UninvertingDirectoryReader())} [junit4] 2> 1128299 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.u.UpdateLog Co= uld not find max version in index or recent updates, using new clock 157198= 9831519567872 [junit4] 2> 1128302 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderE= lectionContext Enough replicas found to continue. [junit4] 2> 1128302 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderE= lectionContext I may be the new leader - try and sync [junit4] 2> 1128302 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.SyncStrategy= Sync replicas to http://127.0.0.1:43481/collection1_shard1_replica_t0/ [junit4] 2> 1128302 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.SyncStrategy= Sync Success - now sync replicas to me [junit4] 2> 1128302 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.SyncStrategy= http://127.0.0.1:43481/collection1_shard1_replica_t0/ has no replicas [junit4] 2> 1128302 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderE= lectionContext Found all replicas participating in election, clear LIR [junit4] 2> 1128302 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.ZkController= collection1_shard1_replica_t0 stopping background replication from leader [junit4] 2> 1128303 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.ShardLeaderE= lectionContext I am the new leader: http://127.0.0.1:43481/collection1_shar= d1_replica_t0/ shard1 [junit4] 2> 1128404 INFO (zkCallback-2407-thread-1-processing-n:127.0= .0.1:43481_) [n:127.0.0.1:43481_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [3]) [junit4] 2> 1128404 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [3]) [junit4] 2> 1128453 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.c.ZkController= I am the leader, no recovery necessary [junit4] 2> 1128455 INFO (qtp1516271500-18042) [n:127.0.0.1:43481_ c:= collection1 s:shard1 x:collection1_shard1_replica_t0] o.a.s.s.HttpSolrCall= [admin] webapp=3Dnull path=3D/admin/cores params=3D{qt=3D/admin/cores&coll= ection.configName=3Dconf1&name=3Dcollection1_shard1_replica_t0&action=3DCRE= ATE&collection=3Dcollection1&shard=3Dshard1&wt=3Djavabin&version=3D2&replic= aType=3DTLOG} status=3D0 QTime=3D1303 [junit4] 2> 1128456 INFO (qtp2071608789-18013) [n:127.0.0.1:38085_ = ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections par= ams=3D{node=3D127.0.0.1:43481_&action=3DADDREPLICA&collection=3Dcollection1= &shard=3Dshard1&type=3DTLOG&wt=3Djavabin&version=3D2} status=3D0 QTime=3D13= 06 [junit4] 2> 1128505 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.AbstractFullDistribZkTe= stBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-mast= er-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsS= afeWithPullReplicasTest_372ABA285F670815-001/shard-3-001 of type PULL [junit4] 2> 1128505 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server jetty-9.3.14.v20= 161028 [junit4] 2> 1128506 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.h.ContextHandler Starte= d o.e.j.s.ServletContextHandler@2ea4468d{/,null,AVAILABLE} [junit4] 2> 1128506 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.AbstractConnector Start= ed ServerConnector@45cff25a{HTTP/1.1,[http/1.1]}{127.0.0.1:46609} [junit4] 2> 1128506 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.e.j.s.Server Started @1129869= ms [junit4] 2> 1128506 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.s.e.JettySolrRunner Jet= ty properties: {solr.data.dir=3D/home/jenkins/workspace/Lucene-Solr-master-= Linux/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafe= WithPullReplicasTest_372ABA285F670815-001/tempDir-001/jetty3, replicaType= =3DPULL, solrconfig=3Dsolrconfig.xml, hostContext=3D/, hostPort=3D46609, co= reRootDirectory=3D/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/bui= ld/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplic= asTest_372ABA285F670815-001/shard-3-001/cores} [junit4] 2> 1128506 ERROR (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.StartupLoggingUtils Mis= sing Java Option solr.log.dir. Logging may be missing or incomplete. [junit4] 2> 1128506 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter ___= _ Welcome to Apache Solr=E2=84=A2 version 8.0.0 [junit4] 2> 1128506 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter / __= | ___| |_ _ Starting in cloud mode on port null [junit4] 2> 1128506 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter \__ = \/ _ \ | '_| Install dir: null, Default config dir: null [junit4] 2> 1128507 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter |___= /\___/_|_| Start time: 2017-07-04T11:05:28.180Z [junit4] 2> 1128508 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.s.SolrDispatchFilter Load= ing solr.xml from SolrHome (not found in ZooKeeper) [junit4] 2> 1128508 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig Loading c= ontainer configuration from /home/jenkins/workspace/Lucene-Solr-master-Linu= x/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeyNothingIsSafeWith= PullReplicasTest_372ABA285F670815-001/shard-3-001/solr.xml [junit4] 2> 1128510 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.SolrXmlConfig MBean ser= ver found: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd, but no JMX repo= rters were configured - adding default JMX reporter. [junit4] 2> 1128512 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [ ] o.a.s.c.ZkContainer Zookeeper c= lient=3D127.0.0.1:33003/solr [junit4] 2> 1128513 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ = ] o.a.z.s.NIOServerCnxn caught end of stream exception [junit4] 2> EndOfStreamException: Unable to read additional data from = client sessionid 0x15d0d46ae0f000e, likely client has closed socket [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOSe= rverCnxn.java:239) [junit4] 2> =09at org.apache.zookeeper.server.NIOServerCnxnFactory.run= (NIOServerCnxnFactory.java:203) [junit4] 2> =09at java.lang.Thread.run(Thread.java:748) [junit4] 2> 1128515 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:46609_ ] o.a.s.c.c.ZkS= tateReader Updated live nodes from ZooKeeper... (0) -> (3) [junit4] 2> 1128515 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:46609_ ] o.a.s.c.Overs= eer Overseer (id=3Dnull) closing [junit4] 2> 1128516 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:46609_ ] o.a.s.c.ZkCon= troller Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46609_ [junit4] 2> 1128516 INFO (zkCallback-2396-thread-1) [ ] o.a.s.c.c.= ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1128516 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1128516 INFO (zkCallback-2413-thread-1-processing-n:127.0= .0.1:46609_) [n:127.0.0.1:46609_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1128516 INFO (zkCallback-2407-thread-1-processing-n:127.0= .0.1:43481_) [n:127.0.0.1:43481_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1128516 INFO (zkCallback-2389-thread-2-processing-n:127.0= .0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.c.ZkStateReader Updated live = nodes from ZooKeeper... (3) -> (4) [junit4] 2> 1128557 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:46609_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled = at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1128560 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:46609_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at= server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1128560 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:46609_ ] o.a.s.m.r.Sol= rJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enable= d at server: com.sun.jmx.mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1128560 INFO (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]) [n:127.0.0.1:46609_ ] o.a.s.c.CoreP= ropertiesLocator Found 0 core definitions underneath /home/jenkins/workspac= e/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.Cha= osMonkeyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/shard-3-001/= cores [junit4] 2> 1128598 INFO (qtp2071608789-18008) [n:127.0.0.1:38085_ = ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with = params node=3D127.0.0.1:46609_&action=3DADDREPLICA&collection=3Dcollection1= &shard=3Dshard2&type=3DPULL&wt=3Djavabin&version=3D2 and sendToOCPQueue=3Dt= rue [junit4] 2> 1128599 INFO (OverseerThreadFactory-6610-thread-5-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.AddReplicaCmd addR= eplica() : { [junit4] 2> "collection":"collection1", [junit4] 2> "node":"127.0.0.1:46609_", [junit4] 2> "shard":"shard2", [junit4] 2> "type":"PULL", [junit4] 2> "operation":"addreplica"} [junit4] 2> 1128599 INFO (OverseerCollectionConfigSetProcessor-982493= 64091109380-127.0.0.1:43977_-n_0000000000) [n:127.0.0.1:43977_ ] o.a.s.c= .OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-00= 00000006 doesn't exist. Requestor may have disconnected from ZooKeeper [junit4] 2> 1128600 INFO (OverseerThreadFactory-6610-thread-5-process= ing-n:127.0.0.1:43977_) [n:127.0.0.1:43977_ ] o.a.s.c.AddReplicaCmd Node= Identified 127.0.0.1:46609_ for creating new replica [junit4] 2> 1128600 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ = ] o.a.s.h.a.CoreAdminOperation core create command qt=3D/admin/cores&collec= tion.configName=3Dconf1&name=3Dcollection1_shard2_replica_p1&action=3DCREAT= E&collection=3Dcollection1&shard=3Dshard2&wt=3Djavabin&version=3D2&replicaT= ype=3DPULL [junit4] 2> 1128601 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ = ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 tr= ansient cores [junit4] 2> 1128703 INFO (zkCallback-2407-thread-1-processing-n:127.0= .0.1:43481_) [n:127.0.0.1:43481_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [4]) [junit4] 2> 1128703 INFO (zkCallback-2401-thread-1-processing-n:127.0= .0.1:38085_) [n:127.0.0.1:38085_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [4]) [junit4] 2> 1128703 INFO (zkCallback-2413-thread-1-processing-n:127.0= .0.1:46609_) [n:127.0.0.1:46609_ ] o.a.s.c.c.ZkStateReader A cluster sta= te change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/col= lections/collection1/state.json] for collection [collection1] has occurred = - updating... (live nodes size: [4]) [junit4] 2> 1129616 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.SolrConfig Us= ing Lucene MatchVersion: 8.0.0 [junit4] 2> 1129631 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.s.IndexSchema [= collection1_shard2_replica_p1] Schema name=3Dtest [junit4] 2> 1129696 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.s.IndexSchema L= oaded schema test/1.6 with uniqueid field id [junit4] 2> 1129702 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.CoreContainer= Creating SolrCore 'collection1_shard2_replica_p1' using configuration from= collection collection1, trusted=3Dtrue [junit4] 2> 1129702 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.m.r.SolrJmxRepo= rter JMX monitoring for 'solr.core.collection1.shard2.replica_p1' (registry= 'solr.core.collection1.shard2.replica_p1') enabled at server: com.sun.jmx.= mbeanserver.JmxMBeanServer@79d72dfd [junit4] 2> 1129702 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.SolrCore solr= .RecoveryStrategy.Builder [junit4] 2> 1129702 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.SolrCore [[co= llection1_shard2_replica_p1] ] Opening new SolrCore at [/home/jenkins/works= pace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.= ChaosMonkeyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/shard-3-0= 01/cores/collection1_shard2_replica_p1], dataDir=3D[/home/jenkins/workspace= /Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.Chao= sMonkeyNothingIsSafeWithPullReplicasTest_372ABA285F670815-001/shard-3-001/c= ores/collection1_shard2_replica_p1/data/] [junit4] 2> 1129704 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.u.RandomMergePo= licy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePo= licy: [TieredMergePolicy: maxMergeAtOnce=3D26, maxMergeAtOnceExplicit=3D50,= maxMergedSegmentMB=3D49.140625, floorSegmentMB=3D0.7548828125, forceMergeD= eletesPctAllowed=3D13.188791359168574, segmentsPerTier=3D23.0, maxCFSSegmen= tSizeMB=3D8.796093022207999E12, noCFSRatio=3D0.0 [junit4] 2> 1129723 WARN (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.RequestHandle= rs INVALID paramSet a in requestHandler {type =3D requestHandler,name =3D /= dump,class =3D DumpRequestHandler,attributes =3D {initParams=3Da, name=3D/d= ump, class=3DDumpRequestHandler},args =3D {defaults=3D{a=3DA,b=3DB}}} [junit4] 2> 1129743 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.u.CommitTracker= Hard AutoCommit: if uncommited for 15000ms;=20 [junit4] 2> 1129743 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.u.CommitTracker= Soft AutoCommit: disabled [junit4] 2> 1129744 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.u.RandomMergePo= licy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePo= licy: [LogDocMergePolicy: minMergeSize=3D1000, mergeFactor=3D24, maxMergeSi= ze=3D9223372036854775807, maxMergeSizeForForcedMerge=3D9223372036854775807,= calibrateSizeByDeletes=3Dtrue, maxMergeDocs=3D2147483647, maxCFSSegmentSiz= eMB=3D8.796093022207999E12, noCFSRatio=3D0.34506825957671705] [junit4] 2> 1129744 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.s.SolrIndexSear= cher Opening [Searcher@6a0606d5[collection1_shard2_replica_p1] main] [junit4] 2> 1129745 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.r.ManagedResour= ceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1 [junit4] 2> 1129746 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.r.ManagedResour= ceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:p= ath=3D/configs/conf1 [junit4] 2> 1129746 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.h.ReplicationHa= ndler Commits will be reserved for 10000 [junit4] 2> 1129746 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.ZkController = collection1_shard2_replica_p1 starting background replication from leader [junit4] 2> 1129746 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.ReplicateFrom= Leader Will start replication from leader with poll interval: 0:0:7 [junit4] 2> 1129747 INFO (searcherExecutor-6646-thread-1-processing-n= :127.0.0.1:46609_ x:collection1_shard2_replica_p1 s:shard2 c:collection1) [= n:127.0.0.1:46609_ c:collection1 s:shard2 x:collection1_shard2_replica_p1]= o.a.s.c.SolrCore [collection1_shard2_replica_p1] Registered new searcher S= earcher@6a0606d5[collection1_shard2_replica_p1] main{ExitableDirectoryReade= r(UninvertingDirectoryReader())} [junit4] 2> 1129747 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.h.ReplicationHa= ndler Poll scheduled at an interval of 7000ms [junit4] 2> 1129747 INFO (indexFetcher-6651-thread-1) [n:127.0.0.1:46= 609_ c:collection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.h.Index= Fetcher Updated masterUrl to http://127.0.0.1:38085/collection1_shard2_repl= ica_t0/ [junit4] 2> 1129747 INFO (qtp502487722-18071) [n:127.0.0.1:46609_ c:c= ollection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.c.ZkController = Core needs to recover:collection1_shard2_replica_p1 [junit4] 2> 1129748 INFO (updateExecutor-2410-thread-1-processing-n:1= 27.0.0.1:46609_ x:collection1_shard2_replica_p1 s:shard2 c:collection1) [n:= 127.0.0.1:46609_ c:collection1 s:shard2 r:core_node3 x:collection1_shard2_r= eplica_p1] o.a.s.u.DefaultSolrCoreState Running recovery [junit4] 2> 1129748 INFO (qtp2071608789-18012) [n:127.0.0.1:38085_ c:= collection1 s:shard2 r:core_node1 x:collection1_shard2_replica_t0] o.a.s.c.= S.Request [collection1_shard2_replica_t0] webapp=3D path=3D/replication pa= rams=3D{qt=3D/replication&wt=3Djavabin&version=3D2&command=3Dindexversion} = status=3D0 QTime=3D0 [junit4] 2> 1129748 INFO (recoveryExecutor-2411-thread-1-processing-n= :127.0.0.1:46609_ x:collection1_shard2_replica_p1 s:shard2 c:collection1 r:= core_node3) [n:127.0.0.1:46609_ c:collection1 s:shard2 r:core_node3 x:colle= ction1_shard2_replica_p1] o.a.s.c.RecoveryStrategy Starting recovery proces= s. recoveringAfterStartup=3Dtrue [junit4] 2> 1129748 INFO (indexFetcher-6651-thread-1) [n:127.0.0.1:46= 609_ c:collection1 s:shard2 x:collection1_shard2_replica_p1] o.a.s.h.Index= Fetcher Master's generation: 1 [junit4] 2> 1129748 INFO (recoveryExecutor-2411-thread-1-processing-n= :127.0.0.1:46609_ x:collection1_shard2_replica_p1 s:shard2 c:collection1 r:= core_node3) [n:127.0.0.1:46609_ c:collection1 s:shard2 r:core_node3 x:colle= ction1_shard2_replica_p1] o.a.s.c.RecoveryStrategy Publishing state of core= [collection1_shard2_replica_p1] as r [...truncated too long message...] o.a.z.c.StaticHostProvider Unexpected exception [junit4] 2> java.lang.InterruptedException: sleep interrupted [junit4] 2> =09at java.lang.Thread.sleep(Native Method) [junit4] 2> =09at org.apache.zookeeper.client.StaticHostProvider.next(= StaticHostProvider.java:101) [junit4] 2> =09at org.apache.zookeeper.ClientCnxn$SendThread.startConn= ect(ClientCnxn.java:997) [junit4] 2> =09at org.apache.zookeeper.ClientCnxn$SendThread.run(Clien= tCnxn.java:1060) [junit4] 2> 1231976 WARN (TEST-ChaosMonkeyNothingIsSafeWithPullReplic= asTest.test-seed#[372ABA285F670815]-SendThread(127.0.0.1:33003)) [ ] o.a= .z.ClientCnxn Session 0x15d0d46ae0f0006 for server null, unexpected error, = closing socket connection and attempting reconnect [junit4] 2> java.net.ConnectException: Connection refused [junit4] 2> =09at sun.nio.ch.SocketChannelImpl.checkConnect(Native Met= hod) [junit4] 2> =09at sun.nio.ch.SocketChannelImpl.finishConnect(SocketCha= nnelImpl.java:717) [junit4] 2> =09at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport= (ClientCnxnSocketNIO.java:361) [junit4] 2> =09at org.apache.zookeeper.ClientCnxn$SendThread.run(Clien= tCnxn.java:1141) [junit4] 2> Jul 04, 2017 11:07:12 AM com.carrotsearch.randomizedtestin= g.ThreadLeakControl tryToInterruptAll [junit4] 2> SEVERE: There are still zombie threads that couldn't be te= rminated: [junit4] 2> 1) Thread[id=3D18003, name=3DTEST-ChaosMonkeyNothingIsS= afeWithPullReplicasTest.test-seed#[372ABA285F670815]-SendThread(127.0.0.1:3= 3003), state=3DTIMED_WAITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPull= ReplicasTest] [junit4] 2> at java.lang.Thread.sleep(Native Method) [junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.= next(StaticHostProvider.java:101) [junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.star= tConnect(ClientCnxn.java:997) [junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(= ClientCnxn.java:1060) [junit4] 2> NOTE: test params are: codec=3DLucene70, sim=3DRandomSimil= arity(queryNorm=3Dfalse): {}, locale=3Des-PR, timezone=3DEurope/Madrid [junit4] 2> NOTE: Linux 4.10.0-21-generic amd64/Oracle Corporation 1.8= .0_131 (64-bit)/cpus=3D8,threads=3D2,free=3D272132288,total=3D518979584 [junit4] 2> NOTE: All tests run in this JVM: [TestInitQParser, XsltUpd= ateRequestHandlerTest, TestPullReplica, BlockCacheTest, TestRecovery, TestU= tils, TestGroupingSearch, HdfsThreadLeakTest, LeaderFailoverAfterPartitionT= est, PreAnalyzedUpdateProcessorTest, TestJsonFacets, MinimalSchemaTest, Tes= tMissingGroups, TestPointFields, TestReversedWildcardFilterFactory, DateMat= hParserTest, TestHttpShardHandlerFactory, BasicDistributedZk2Test, TestTrie= , TestSubQueryTransformerDistrib, RequestLoggingTest, CoreSorterTest, SolrT= estCaseJ4Test, TestJmxIntegration, TestFieldCache, TestSolrXml, HdfsCollect= ionsAPIDistributedZkTest, TestFieldTypeResource, TestUninvertingReader, Tes= tSolrCloudWithDelegationTokens, TestFoldingMultitermQuery, TestRawResponseW= riter, TestNumericRangeQuery32, DeleteInactiveReplicaTest, TestDistributedM= issingSort, SimplePostToolTest, OverseerModifyCollectionTest, BasicAuthStan= daloneTest, TestRestManager, ClassificationUpdateProcessorIntegrationTest, = DocValuesMissingTest, SolrCloudExampleTest, DistribCursorPagingTest, TestSu= rroundQueryParser, RecoveryZkTest, TestBulkSchemaConcurrent, TestComplexPhr= aseQParserPlugin, SearchHandlerTest, SolrMetricsIntegrationTest, TestFilter= edDocIdSet, TestReload, TestMiniSolrCloudClusterSSL, OutOfBoxZkACLAndCreden= tialsProvidersTest, RemoteQueryErrorTest, UnloadDistributedZkTest, ChangedS= chemaMergeTest, TestApiFramework, HardAutoCommitTest, CdcrReplicationHandle= rTest, TestRTimerTree, TestHdfsUpdateLog, RegexBoostProcessorTest, TestClou= dPivotFacet, OverseerTest, TestNamedUpdateProcessors, PrimitiveFieldTypeTes= t, DirectUpdateHandlerOptimizeTest, AutoScalingHandlerTest, DistributedQueu= eTest, MergeStrategyTest, TestReqParamsAPI, TimeZoneUtilsTest, SolrJmxRepor= terTest, RequiredFieldsTest, TestSearcherReuse, LeaderElectionTest, CheckHd= fsIndexTest, MoveReplicaTest, URLClassifyProcessorTest, TestJettySolrRunner= , DistributedFacetExistsSmallTest, FullSolrCloudDistribCmdsTest, BaseCdcrDi= stributedZkTest, TestCloudNestedDocsSort, TestConfigOverlay, BigEndianAscen= dingWordSerializerTest, TestAnalyzedSuggestions, CircularListTest, TestStre= ssCloudBlindAtomicUpdates, NotRequiredUniqueKeyTest, TestLegacyFieldCache, = SolrCoreCheckLockOnStartupTest, TestLFUCache, FullHLLTest, MultiTermTest, T= estStandardQParsers, ReturnFieldsTest, ShufflingReplicaListTransformerTest,= ZkStateReaderTest, TestCopyFieldCollectionResource, ShowFileRequestHandler= Test, SolrCoreMetricManagerTest, BlockJoinFacetRandomTest, SolrInfoBeanTest= , PreAnalyzedFieldTest, TestOverriddenPrefixQueryForCustomFieldType, Collec= tionsAPISolrJTest, CopyFieldTest, HdfsRecoveryZkTest, TestDefaultStatsCache= , NumericFieldsTest, SpellCheckCollatorWithCollapseTest, TestXmlQParser, Te= stCloudDeleteByQuery, TestStressReorder, TlogReplayBufferedWhileIndexingTes= t, LeaderElectionIntegrationTest, ClusterStateUpdateTest, ZkSolrClientTest,= ZkCLITest, AutoCommitTest, TestCoreContainer, SimpleFacetsTest, SolrCoreTe= st, SpellCheckComponentTest, QueryElevationComponentTest, PeerSyncTest, Tes= tIndexSearcher, DistributedQueryElevationComponentTest, TestCoreDiscovery, = SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SuggesterFSTTe= st, CoreAdminHandlerTest, DocValuesTest, SuggesterTest, TestStressLucene, P= olyFieldTest, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, TestPse= udoReturnFields, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCas= es, TestWordDelimiterFilterFactory, QueryEqualityTest, StatelessScriptUpdat= eProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, DistanceFun= ctionTest, TestSolrDeletionPolicy1, CacheHeaderTest, IndexBasedSpellChecker= Test, LukeRequestHandlerTest, TestQueryUtils, TestWriterPerf, DirectSolrSpe= llCheckerTest, FileBasedSpellCheckerTest, TermsComponentTest, DocumentBuild= erTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, MoreLikeTh= isHandlerTest, FastVectorHighlighterTest, SolrPluginUtilsTest, MBeansHandle= rTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestMergePol= icyConfig, TestSolrDeletionPolicy2, TestElisionMultitermQuery, OutputWriter= Test, TestSolrCoreProperties, TestPostingsSolrHighlighter, DirectSolrConnec= tionTest, SpellPossibilityIteratorTest, TestCodecSupport, TestXIncludeConfi= g, EchoParamsTest, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, = TestLMJelinekMercerSimilarityFactory, TestFastWriter, TestFastOutputStream,= ChaosMonkeyNothingIsSafeTest, CdcrReplicationDistributedZkTest, CdcrVersio= nReplicationTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=3DChaosMonkeyNo= thingIsSafeWithPullReplicasTest -Dtests.seed=3D372ABA285F670815 -Dtests.mul= tiplier=3D3 -Dtests.slow=3Dtrue -Dtests.locale=3Des-PR -Dtests.timezone=3DE= urope/Madrid -Dtests.asserts=3Dtrue -Dtests.file.encoding=3DUTF-8 [junit4] ERROR 0.00s J2 | ChaosMonkeyNothingIsSafeWithPullReplicasTest= (suite) <<< [junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLea= kError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMon= keyNothingIsSafeWithPullReplicasTest:=20 [junit4] > 1) Thread[id=3D18002, name=3DConnection evictor, state= =3DTIMED_WAITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest= ] [junit4] > at java.lang.Thread.sleep(Native Method) [junit4] > at org.apache.http.impl.client.IdleConnectionEvict= or$1.run(IdleConnectionEvictor.java:66) [junit4] > at java.lang.Thread.run(Thread.java:748) [junit4] > 2) Thread[id=3D18003, name=3DTEST-ChaosMonkeyNothingIsS= afeWithPullReplicasTest.test-seed#[372ABA285F670815]-SendThread(127.0.0.1:3= 3003), state=3DTIMED_WAITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPull= ReplicasTest] [junit4] > at java.lang.Thread.sleep(Native Method) [junit4] > at org.apache.zookeeper.client.StaticHostProvider.= next(StaticHostProvider.java:101) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.star= tConnect(ClientCnxn.java:997) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(= ClientCnxn.java:1060) [junit4] > 3) Thread[id=3D18004, name=3DTEST-ChaosMonkeyNothingIsS= afeWithPullReplicasTest.test-seed#[372ABA285F670815]-EventThread, state=3DW= AITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] [junit4] > at sun.misc.Unsafe.park(Native Method) [junit4] > at java.util.concurrent.locks.LockSupport.park(Loc= kSupport.java:175) [junit4] > at java.util.concurrent.locks.AbstractQueuedSynchr= onizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) [junit4] > at java.util.concurrent.LinkedBlockingQueue.take(L= inkedBlockingQueue.java:442) [junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run= (ClientCnxn.java:501) [junit4] > =09at __randomizedtesting.SeedInfo.seed([372ABA285F670815]= :0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There = are still zombie threads that couldn't be terminated: [junit4] > 1) Thread[id=3D18003, name=3DTEST-ChaosMonkeyNothingIsS= afeWithPullReplicasTest.test-seed#[372ABA285F670815]-SendThread(127.0.0.1:3= 3003), state=3DTIMED_WAITING, group=3DTGRP-ChaosMonkeyNothingIsSafeWithPull= ReplicasTest] [junit4] > at java.lang.Thread.sleep(Native Method) [junit4] > at org.apache.zookeeper.client.StaticHostProvider.= next(StaticHostProvider.java:101) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.star= tConnect(ClientCnxn.java:997) [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(= ClientCnxn.java:1060) [junit4] > =09at __randomizedtesting.SeedInfo.seed([372ABA285F670815]= :0) [junit4] Completed [397/725 (1!)] on J2 in 109.98s, 1 test, 3 errors <<<= FAILURES! [...truncated 44304 lines...] ------=_Part_296_928765754.1499168080548 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org ------=_Part_296_928765754.1499168080548--