Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id AC067200BB4 for ; Tue, 1 Nov 2016 15:47:39 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id AA33E160AF7; Tue, 1 Nov 2016 14:47:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 088FF160AE5 for ; Tue, 1 Nov 2016 15:47:36 +0100 (CET) Received: (qmail 39839 invoked by uid 500); 1 Nov 2016 14:47:35 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 39825 invoked by uid 99); 1 Nov 2016 14:47:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Nov 2016 14:47:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 15012C0C04 for ; Tue, 1 Nov 2016 14:47:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.001 X-Spam-Level: ** X-Spam-Status: No, score=2.001 tagged_above=-999 required=6.31 tests=[KAM_BADIPHTTP=2, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Bjmoft2lRFTh for ; Tue, 1 Nov 2016 14:47:19 +0000 (UTC) Received: from serv1.sd-datasolutions.de (serv1.sd-datasolutions.de [188.138.57.78]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 9C39C5F2C3 for ; Tue, 1 Nov 2016 14:47:18 +0000 (UTC) Received: from serv1 (localhost.localdomain [127.0.0.1]) by serv1.sd-datasolutions.de (Postfix) with ESMTP id 0FADA4C0591; Tue, 1 Nov 2016 14:47:16 +0000 (UTC) Date: Tue, 1 Nov 2016 14:45:40 +0000 (UTC) From: Policeman Jenkins Server To: shalin@apache.org, dev@lucene.apache.org Message-ID: <1463479887.179.1478011636065.JavaMail.jenkins@serv1> In-Reply-To: <1566865817.171.1477970675449.JavaMail.jenkins@serv1> References: <1566865817.171.1477970675449.JavaMail.jenkins@serv1> Subject: [JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 486 - Still Unstable! MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_178_309502453.1478011636011" X-Jenkins-Job: Lucene-Solr-6.x-Solaris X-Jenkins-Result: UNSTABLE archived-at: Tue, 01 Nov 2016 14:47:39 -0000 ------=_Part_178_309502453.1478011636011 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/486/ Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZ= kTest Error Message: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionL= og] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException= at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTr= acker.java:43) at org.apache.solr.update.HdfsTransactionLog.(HdfsTra= nsactionLog.java:130) at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpd= ateLog.java:202) at org.apache.solr.update.UpdateHandler.(UpdateHand= ler.java:137) at org.apache.solr.update.UpdateHandler.(UpdateHandler= .java:94) at org.apache.solr.update.DirectUpdateHandler2.(DirectUpda= teHandler2.java:102) at sun.reflect.GeneratedConstructorAccessor169.newIns= tance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.new= Instance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.C= onstructor.newInstance(Constructor.java:423) at org.apache.solr.core.SolrC= ore.createInstance(SolrCore.java:706) at org.apache.solr.core.SolrCore.cre= ateUpdateHandler(SolrCore.java:768) at org.apache.solr.core.SolrCore.initU= pdateHandler(SolrCore.java:1007) at org.apache.solr.core.SolrCore.(S= olrCore.java:872) at org.apache.solr.core.SolrCore.(SolrCore.java:77= 6) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:842) a= t org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498) = at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache= .solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(= ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorke= r(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor= $Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.ja= va:745) =20 Stack Trace: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not rel= eased!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException =09at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseT= racker.java:43) =09at org.apache.solr.update.HdfsTransactionLog.(HdfsTransactionLog.j= ava:130) =09at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:202) =09at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:137) =09at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:94) =09at org.apache.solr.update.DirectUpdateHandler2.(DirectUpdateHandle= r2.java:102) =09at sun.reflect.GeneratedConstructorAccessor169.newInstance(Unknown Sourc= e) =09at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC= onstructorAccessorImpl.java:45) =09at java.lang.reflect.Constructor.newInstance(Constructor.java:423) =09at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:706) =09at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:768) =09at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1007) =09at org.apache.solr.core.SolrCore.(SolrCore.java:872) =09at org.apache.solr.core.SolrCore.(SolrCore.java:776) =09at org.apache.solr.core.CoreContainer.create(CoreContainer.java:842) =09at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:4= 98) =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.l= ambda$execute$0(ExecutorUtil.java:229) =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1142) =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:617) =09at java.lang.Thread.run(Thread.java:745) =09at __randomizedtesting.SeedInfo.seed([3FFC99BF5C01CF22]:0) =09at org.junit.Assert.fail(Assert.java:93) =09at org.junit.Assert.assertTrue(Assert.java:43) =09at org.junit.Assert.assertNull(Assert.java:551) =09at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:= 260) =09at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43) =09at java.lang.reflect.Method.invoke(Method.java:498) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1713) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:870) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:57) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:41) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:53) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:47) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:64) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:54) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:367) =09at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 12307 lines...] [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest [junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-So= lr-6.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecove= ryZkTest_3FFC99BF5C01CF22-001/init-core-data-001 [junit4] 2> 2758254 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and client= Auth (false) via: @org.apache.solr.util.RandomizeSSL(reason=3D, value=3DNaN= , ssl=3DNaN, clientAuth=3DNaN) [junit4] 2> 2758254 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 ser= vers in /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/s= olr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_3FFC99BF5C01CF22-0= 01/tempDir-001 [junit4] 2> 2758255 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER [junit4] 2> 2758258 INFO (Thread-7100) [ ] o.a.s.c.ZkTestServer cl= ient port:0.0.0.0/0.0.0.0:0 [junit4] 2> 2758258 INFO (Thread-7100) [ ] o.a.s.c.ZkTestServer St= arting server [junit4] 2> 2758356 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:43493 [junit4] 2> 2758364 INFO (jetty-launcher-4720-thread-1) [ ] o.e.j.= s.Server jetty-9.3.8.v20160314 [junit4] 2> 2758364 INFO (jetty-launcher-4720-thread-2) [ ] o.e.j.= s.Server jetty-9.3.8.v20160314 [junit4] 2> 2758368 INFO (jetty-launcher-4720-thread-1) [ ] o.e.j.= s.h.ContextHandler Started o.e.j.s.ServletContextHandler@186e428f{/solr,nul= l,AVAILABLE} [junit4] 2> 2758368 INFO (jetty-launcher-4720-thread-2) [ ] o.e.j.= s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1af68462{/solr,nul= l,AVAILABLE} [junit4] 2> 2758371 INFO (jetty-launcher-4720-thread-2) [ ] o.e.j.= s.ServerConnector Started ServerConnector@6e29b8cb{HTTP/1.1,[http/1.1]}{127= .0.0.1:44741} [junit4] 2> 2758371 INFO (jetty-launcher-4720-thread-2) [ ] o.e.j.= s.Server Started @2763368ms [junit4] 2> 2758371 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= c.s.e.JettySolrRunner Jetty properties: {hostContext=3D/solr, hostPort=3D44= 741} [junit4] 2> 2758372 ERROR (jetty-launcher-4720-thread-2) [ ] o.a.s.= s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be miss= ing or incomplete. [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-1) [ ] o.e.j.= s.ServerConnector Started ServerConnector@375735bc{HTTP/1.1,[http/1.1]}{127= .0.0.1:55868} [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= s.SolrDispatchFilter ___ _ Welcome to Apache Solr=E2=84=A2 vers= ion 6.4.0 [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-1) [ ] o.e.j.= s.Server Started @2763369ms [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= c.s.e.JettySolrRunner Jetty properties: {hostContext=3D/solr, hostPort=3D55= 868} [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-11-01T14:21:57.226Z [junit4] 2> 2758372 ERROR (jetty-launcher-4720-thread-1) [ ] o.a.s.= s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be miss= ing or incomplete. [junit4] 2> 2758372 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= s.SolrDispatchFilter ___ _ Welcome to Apache Solr=E2=84=A2 vers= ion 6.4.0 [junit4] 2> 2758373 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null [junit4] 2> 2758373 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null [junit4] 2> 2758373 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-11-01T14:21:57.227Z [junit4] 2> 2758375 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 2758376 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading... [junit4] 2> 2758386 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: s= ocketTimeout=3D340000&connTimeout=3D45000&retry=3Dtrue [junit4] 2> 2758386 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: s= ocketTimeout=3D340000&connTimeout=3D45000&retry=3Dtrue [junit4] 2> 2758387 INFO (jetty-launcher-4720-thread-2) [ ] o.a.s.= c.ZkContainer Zookeeper client=3D127.0.0.1:43493/solr [junit4] 2> 2758388 INFO (jetty-launcher-4720-thread-1) [ ] o.a.s.= c.ZkContainer Zookeeper client=3D127.0.0.1:43493/solr [junit4] 2> 2758436 INFO (jetty-launcher-4720-thread-1) [n:127.0.0.1:= 55868_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader= 127.0.0.1:55868_solr [junit4] 2> 2758437 INFO (jetty-launcher-4720-thread-2) [n:127.0.0.1:= 44741_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/li= ve_nodes/127.0.0.1:44741_solr [junit4] 2> 2758438 INFO (jetty-launcher-4720-thread-1) [n:127.0.0.1:= 55868_solr ] o.a.s.c.Overseer Overseer (id=3D96862871035183110-127.0.0.1= :55868_solr-n_0000000000) starting [junit4] 2> 2758441 INFO (zkCallback-4731-thread-1-processing-n:127.0= .0.1:55868_solr) [n:127.0.0.1:55868_solr ] o.a.s.c.c.ZkStateReader Updat= ed live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 2758441 INFO (zkCallback-4730-thread-1-processing-n:127.0= .0.1:44741_solr) [n:127.0.0.1:44741_solr ] o.a.s.c.c.ZkStateReader Updat= ed live nodes from ZooKeeper... (0) -> (1) [junit4] 2> 2758457 INFO (jetty-launcher-4720-thread-1) [n:127.0.0.1:= 55868_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/li= ve_nodes/127.0.0.1:55868_solr [junit4] 2> 2758460 INFO (zkCallback-4730-thread-1-processing-n:127.0= .0.1:44741_solr) [n:127.0.0.1:44741_solr ] o.a.s.c.c.ZkStateReader Updat= ed live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 2758460 INFO (zkCallback-4731-thread-1-processing-n:127.0= .0.1:55868_solr) [n:127.0.0.1:55868_solr ] o.a.s.c.c.ZkStateReader Updat= ed live nodes from ZooKeeper... (1) -> (2) [junit4] 2> 2758475 INFO (jetty-launcher-4720-thread-2) [n:127.0.0.1:= 44741_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions unde= rneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/so= lr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_3FFC99BF5C01CF22-00= 1/tempDir-001/node2/. [junit4] 2> 2758485 INFO (jetty-launcher-4720-thread-1) [n:127.0.0.1:= 55868_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions unde= rneath /export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/so= lr-core/test/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_3FFC99BF5C01CF22-00= 1/tempDir-001/node1/. [junit4] 2> 2758495 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKee= per... (0) -> (2) [junit4] 1> Formatting using clusterid: testClusterID [junit4] 2> 2758540 WARN (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: t= ried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties [junit4] 2> 2758547 WARN (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be e= nabled using Log4j [junit4] 2> 2758550 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 2758567 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/ca= che/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapp= s/hdfs to ./temp/Jetty_solaris.vm_39486_hdfs____pys59a/webapp [junit4] 2> 2759058 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWith= SafeStartup@solaris-vm:39486 [junit4] 2> 2759179 WARN (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be e= nabled using Log4j [junit4] 2> 2759180 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 2759193 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/ca= che/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapp= s/datanode to ./temp/Jetty_localhost_32780_datanode____ah779z/webapp [junit4] 2> 2759705 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWith= SafeStartup@localhost:32780 [junit4] 2> 2759781 WARN (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be e= nabled using Log4j [junit4] 2> 2759782 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log jetty-6.1.26 [junit4] 2> 2759798 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log Extract jar:file:/export/home/jenkins/.ivy2/ca= che/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapp= s/datanode to ./temp/Jetty_localhost_54706_datanode____53hht1/webapp [junit4] 2> 2759864 INFO (IPC Server handler 5 on 42961) [ ] Block= StateChange BLOCK* processReport: from storage DS-36d3a1a6-7e2c-4849-b88c-1= d87a7ac2fdb node DatanodeRegistration(127.0.0.1:48326, datanodeUuid=3D13ba2= b77-8e5d-4bd0-8f23-b05a1cd8b8cb, infoPort=3D53250, infoSecurePort=3D0, ipcP= ort=3D46219, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D1779051302;c= =3D0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs [junit4] 2> 2759864 INFO (IPC Server handler 5 on 42961) [ ] Block= StateChange BLOCK* processReport: from storage DS-19464f9a-5c1f-4676-b2d8-4= bd248b2460a node DatanodeRegistration(127.0.0.1:48326, datanodeUuid=3D13ba2= b77-8e5d-4bd0-8f23-b05a1cd8b8cb, infoPort=3D53250, infoSecurePort=3D0, ipcP= ort=3D46219, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D1779051302;c= =3D0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs [junit4] 2> 2760355 INFO (SUITE-HdfsRecoveryZkTest-seed#[3FFC99BF5C01= CF22]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWith= SafeStartup@localhost:54706 [junit4] 2> 2760486 INFO (IPC Server handler 7 on 42961) [ ] Block= StateChange BLOCK* processReport: from storage DS-9f4aeaf4-eaaa-44e3-98e3-e= 38d0c66280f node DatanodeRegistration(127.0.0.1:48964, datanodeUuid=3Db5b51= 266-ef04-4d25-8ae4-de028ebcdf79, infoPort=3D47650, infoSecurePort=3D0, ipcP= ort=3D58525, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D1779051302;c= =3D0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs [junit4] 2> 2760486 INFO (IPC Server handler 7 on 42961) [ ] Block= StateChange BLOCK* processReport: from storage DS-d2bc052d-64f8-4663-911e-5= 58bbf743f02 node DatanodeRegistration(127.0.0.1:48964, datanodeUuid=3Db5b51= 266-ef04-4d25-8ae4-de028ebcdf79, infoPort=3D47650, infoSecurePort=3D0, ipcP= ort=3D58525, storageInfo=3Dlv=3D-56;cid=3DtestClusterID;nsid=3D1779051302;c= =3D0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs [junit4] 2> 2760653 INFO (TEST-HdfsRecoveryZkTest.test-seed#[3FFC99BF= 5C01CF22]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test [junit4] 2> 2760656 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with p= arams replicationFactor=3D2&collection.configName=3Dconf&maxShardsPerNode= =3D1&name=3Drecoverytest&action=3DCREATE&numShards=3D1&wt=3Djavabin&version= =3D2 and sendToOCPQueue=3Dtrue [junit4] 2> 2760659 INFO (OverseerThreadFactory-12459-thread-1-proces= sing-n:127.0.0.1:55868_solr) [n:127.0.0.1:55868_solr ] o.a.s.c.CreateCol= lectionCmd Create collection recoverytest [junit4] 2> 2760769 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= ] o.a.s.h.a.CoreAdminOperation core create command qt=3D/admin/cores&co= llection.configName=3Dconf&newCollection=3Dtrue&name=3Drecoverytest_shard1_= replica1&action=3DCREATE&numShards=3D1&collection=3Drecoverytest&shard=3Dsh= ard1&wt=3Djavabin&version=3D2 [junit4] 2> 2760769 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r ] o.a.s.h.a.CoreAdminOperation core create command qt=3D/admin/cores&c= ollection.configName=3Dconf&newCollection=3Dtrue&name=3Drecoverytest_shard1= _replica2&action=3DCREATE&numShards=3D1&collection=3Drecoverytest&shard=3Ds= hard1&wt=3Djavabin&version=3D2 [junit4] 2> 2760882 INFO (zkCallback-4731-thread-1-processing-n:127.0= .0.1:55868_solr) [n:127.0.0.1:55868_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2760882 INFO (zkCallback-4730-thread-1-processing-n:127.0= .0.1:44741_solr) [n:127.0.0.1:44741_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2761783 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.SolrConfi= g Using Lucene MatchVersion: 6.4.0 [junit4] 2> 2761783 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.c.SolrConf= ig Using Lucene MatchVersion: 6.4.0 [junit4] 2> 2761812 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.s.IndexSch= ema [recoverytest_shard1_replica2] Schema name=3Dminimal [junit4] 2> 2761812 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.s.IndexSche= ma [recoverytest_shard1_replica1] Schema name=3Dminimal [junit4] 2> 2761815 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.s.IndexSch= ema Loaded schema minimal/1.1 with uniqueid field id [junit4] 2> 2761815 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.s.IndexSche= ma Loaded schema minimal/1.1 with uniqueid field id [junit4] 2> 2761817 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 x:recoverytest_shard1_replica2] o.a.s.c.CoreCont= ainer Creating SolrCore 'recoverytest_shard1_replica2' using configuration = from collection recoverytest [junit4] 2> 2761817 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 x:recoverytest_shard1_replica1] o.a.s.c.CoreConta= iner Creating SolrCore 'recoverytest_shard1_replica1' using configuration f= rom collection recoverytest [junit4] 2> 2761824 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory solr.hdfs.home=3Dhdfs://solaris-vm:42961/data [junit4] 2> 2761824 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled [junit4] 2> 2761824 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.SolrCore [[recoverytest_shard1_replica2] ] Opening new SolrCore at [/ex= port/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/te= st/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_3FFC99BF5C01CF22-001/tempDir-= 001/node1/recoverytest_shard1_replica2], dataDir=3D[hdfs://solaris-vm:42961= /data/recoverytest/core_node1/data/] [junit4] 2> 2761824 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris= -vm:42961/data/recoverytest/core_node1/data/snapshot_metadata [junit4] 2> 2761824 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory solr.hdfs.home=3Dhdfs://solaris-vm:42961/data [junit4] 2> 2761824 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory Solr Kerberos Authentication disabled [junit4] 2> 2761824 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.SolrCore [[recoverytest_shard1_replica1] ] Opening new SolrCore at [/exp= ort/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/tes= t/J1/temp/solr.cloud.hdfs.HdfsRecoveryZkTest_3FFC99BF5C01CF22-001/tempDir-0= 01/node2/recoverytest_shard1_replica1], dataDir=3D[hdfs://solaris-vm:42961/= data/recoverytest/core_node2/data/] [junit4] 2> 2761824 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-= vm:42961/data/recoverytest/core_node2/data/snapshot_metadata [junit4] 2> 2761835 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memo= ry allocation set to [true] [junit4] 2> 2761835 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory Block cache target memory usage, slab size of [1677= 7216] will allocate [1] slabs and use ~[16777216] bytes [junit4] 2> 2761835 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache [junit4] 2> 2761837 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct mem= ory allocation set to [true] [junit4] 2> 2761837 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [167= 77216] will allocate [1] slabs and use ~[16777216] bytes [junit4] 2> 2761837 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache [junit4] 2> 2761985 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .s.b.BlockDirectory Block cache on write is disabled [junit4] 2> 2761986 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-= vm:42961/data/recoverytest/core_node2/data [junit4] 2> 2761988 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.s.b.BlockDirectory Block cache on write is disabled [junit4] 2> 2761989 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris= -vm:42961/data/recoverytest/core_node1/data [junit4] 2> 2762006 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris-= vm:42961/data/recoverytest/core_node2/data/index [junit4] 2> 2762008 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory creating directory factory for path hdfs://solaris= -vm:42961/data/recoverytest/core_node1/data/index [junit4] 2> 2762014 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memo= ry allocation set to [true] [junit4] 2> 2762014 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory Block cache target memory usage, slab size of [1677= 7216] will allocate [1] slabs and use ~[16777216] bytes [junit4] 2> 2762014 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache [junit4] 2> 2762015 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct mem= ory allocation set to [true] [junit4] 2> 2762015 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [167= 77216] will allocate [1] slabs and use ~[16777216] bytes [junit4] 2> 2762015 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache [junit4] 2> 2762086 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.s.b.BlockDirectory Block cache on write is disabled [junit4] 2> 2762087 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .s.b.BlockDirectory Block cache on write is disabled [junit4] 2> 2762169 INFO (IPC Server handler 7 on 42961) [ ] Block= StateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48326 is add= ed to blk_1073741826_1002{UCState=3DUNDER_CONSTRUCTION, truncateBlock=3Dnul= l, primaryNodeIndex=3D-1, replicas=3D[ReplicaUC[[DISK]DS-9f4aeaf4-eaaa-44e3= -98e3-e38d0c66280f:NORMAL:127.0.0.1:48964|RBW], ReplicaUC[[DISK]DS-36d3a1a6= -7e2c-4849-b88c-1d87a7ac2fdb:NORMAL:127.0.0.1:48326|FINALIZED]]} size 0 [junit4] 2> 2762170 INFO (IPC Server handler 4 on 42961) [ ] Block= StateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48964 is add= ed to blk_1073741825_1001{UCState=3DUNDER_CONSTRUCTION, truncateBlock=3Dnul= l, primaryNodeIndex=3D-1, replicas=3D[ReplicaUC[[DISK]DS-d2bc052d-64f8-4663= -911e-558bbf743f02:NORMAL:127.0.0.1:48964|RBW], ReplicaUC[[DISK]DS-19464f9a= -5c1f-4676-b2d8-4bd248b2460a:NORMAL:127.0.0.1:48326|RBW]]} size 0 [junit4] 2> 2762170 INFO (IPC Server handler 9 on 42961) [ ] Block= StateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48326 is add= ed to blk_1073741825_1001{UCState=3DUNDER_CONSTRUCTION, truncateBlock=3Dnul= l, primaryNodeIndex=3D-1, replicas=3D[ReplicaUC[[DISK]DS-d2bc052d-64f8-4663= -911e-558bbf743f02:NORMAL:127.0.0.1:48964|RBW], ReplicaUC[[DISK]DS-19464f9a= -5c1f-4676-b2d8-4bd248b2460a:NORMAL:127.0.0.1:48326|RBW]]} size 0 [junit4] 2> 2762171 INFO (IPC Server handler 0 on 42961) [ ] Block= StateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48964 is add= ed to blk_1073741826_1002{UCState=3DUNDER_CONSTRUCTION, truncateBlock=3Dnul= l, primaryNodeIndex=3D-1, replicas=3D[ReplicaUC[[DISK]DS-9f4aeaf4-eaaa-44e3= -98e3-e38d0c66280f:NORMAL:127.0.0.1:48964|RBW], ReplicaUC[[DISK]DS-36d3a1a6= -7e2c-4849-b88c-1d87a7ac2fdb:NORMAL:127.0.0.1:48326|FINALIZED]]} size 0 [junit4] 2> 2762186 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.Hdf= sUpdateLog [junit4] 2> 2762186 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.UpdateLog Initializing UpdateLog: dataDir=3Dnull defaultSyncLevel=3DFLUS= H numRecordsToKeep=3D100 maxNumLogsToKeep=3D10 numVersionBuckets=3D65536 [junit4] 2> 2762187 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3D3 [junit4] 2> 2762194 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.Hd= fsUpdateLog [junit4] 2> 2762194 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.UpdateLog Initializing UpdateLog: dataDir=3Dnull defaultSyncLevel=3DFLU= SH numRecordsToKeep=3D100 maxNumLogsToKeep=3D10 numVersionBuckets=3D65536 [junit4] 2> 2762194 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3D3 [junit4] 2> 2762200 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 2762200 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 2762207 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.CommitTracker Hard AutoCommit: disabled [junit4] 2> 2762207 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.CommitTracker Soft AutoCommit: disabled [junit4] 2> 2762229 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .s.SolrIndexSearcher Opening [Searcher@7b30013[recoverytest_shard1_replica1= ] main] [junit4] 2> 2762229 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.s.SolrIndexSearcher Opening [Searcher@81b4b84[recoverytest_shard1_replica= 2] main] [junit4] 2> 2762231 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /co= nfigs/conf [junit4] 2> 2762231 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /c= onfigs/conf [junit4] 2> 2762232 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooK= eeperStorageIO:path=3D/configs/conf [junit4] 2> 2762232 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 2762232 INFO (searcherExecutor-12465-thread-1-processing-= n:127.0.0.1:44741_solr x:recoverytest_shard1_replica1 s:shard1 c:recoveryte= st r:core_node2) [n:127.0.0.1:44741_solr c:recoverytest s:shard1 r:core_nod= e2 x:recoverytest_shard1_replica1] o.a.s.c.SolrCore [recoverytest_shard1_re= plica1] Registered new searcher Searcher@7b30013[recoverytest_shard1_replic= a1] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 2762232 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.UpdateLog Could not find max version in index or recent updates, using n= ew clock 1549805940727873536 [junit4] 2> 2762232 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using Zoo= KeeperStorageIO:path=3D/configs/conf [junit4] 2> 2762232 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.h.ReplicationHandler Commits will be reserved for 10000 [junit4] 2> 2762233 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.UpdateLog Could not find max version in index or recent updates, using = new clock 1549805940728922112 [junit4] 2> 2762236 INFO (searcherExecutor-12464-thread-1-processing-= n:127.0.0.1:55868_solr x:recoverytest_shard1_replica2 s:shard1 c:recoveryte= st r:core_node1) [n:127.0.0.1:55868_solr c:recoverytest s:shard1 r:core_nod= e1 x:recoverytest_shard1_replica2] o.a.s.c.SolrCore [recoverytest_shard1_re= plica2] Registered new searcher Searcher@81b4b84[recoverytest_shard1_replic= a2] main{ExitableDirectoryReader(UninvertingDirectoryReader())} [junit4] 2> 2762248 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.ShardLeaderElectionContext Enough replicas found to continue. [junit4] 2> 2762248 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.ShardLeaderElectionContext I may be the new leader - try and sync [junit4] 2> 2762248 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.SyncStrategy Sync replicas to http://127.0.0.1:44741/solr/recoverytest_s= hard1_replica1/ [junit4] 2> 2762249 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.PeerSync PeerSync: core=3Drecoverytest_shard1_replica1 url=3Dhttp://127.= 0.0.1:44741/solr START replicas=3D[http://127.0.0.1:55868/solr/recoverytest= _shard1_replica2/] nUpdates=3D100 [junit4] 2> 2762249 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.PeerSync PeerSync: core=3Drecoverytest_shard1_replica1 url=3Dhttp://127.= 0.0.1:44741/solr DONE. We have no versions. sync failed. [junit4] 2> 2762252 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.c.S.Request [recoverytest_shard1_replica2] webapp=3D/solr path=3D/get pa= rams=3D{distrib=3Dfalse&qt=3D/get&fingerprint=3Dfalse&getVersions=3D100&wt= =3Djavabin&version=3D2} status=3D0 QTime=3D1 [junit4] 2> 2762253 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.SyncStrategy Leader's attempt to sync with shard failed, moving to the n= ext candidate [junit4] 2> 2762253 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.ShardLeaderElectionContext We failed sync, but we have no versions - we = can't sync in that case - we were active before, so become leader anyway [junit4] 2> 2762259 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44741/s= olr/recoverytest_shard1_replica1/ shard1 [junit4] 2> 2762363 INFO (zkCallback-4731-thread-1-processing-n:127.0= .0.1:55868_solr) [n:127.0.0.1:55868_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2762363 INFO (zkCallback-4730-thread-1-processing-n:127.0= .0.1:44741_solr) [n:127.0.0.1:44741_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2762411 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .c.ZkController I am the leader, no recovery necessary [junit4] 2> 2762414 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{qt=3D/a= dmin/cores&collection.configName=3Dconf&newCollection=3Dtrue&name=3Drecover= ytest_shard1_replica1&action=3DCREATE&numShards=3D1&collection=3Drecoveryte= st&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTime=3D1645 [junit4] 2> 2762517 INFO (zkCallback-4731-thread-1-processing-n:127.0= .0.1:55868_solr) [n:127.0.0.1:55868_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2762517 INFO (zkCallback-4730-thread-1-processing-n:127.0= .0.1:44741_solr) [n:127.0.0.1:44741_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2763255 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/cores params=3D{qt=3D/= admin/cores&collection.configName=3Dconf&newCollection=3Dtrue&name=3Drecove= rytest_shard1_replica2&action=3DCREATE&numShards=3D1&collection=3Drecoveryt= est&shard=3Dshard1&wt=3Djavabin&version=3D2} status=3D0 QTime=3D2486 [junit4] 2> 2763264 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for= at most 30 seconds. Check all shard replicas [junit4] 2> 2763366 INFO (zkCallback-4731-thread-2-processing-n:127.0= .0.1:55868_solr) [n:127.0.0.1:55868_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2763366 INFO (zkCallback-4730-thread-1-processing-n:127.0= .0.1:44741_solr) [n:127.0.0.1:44741_solr ] o.a.s.c.c.ZkStateReader A clu= ster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged p= ath:/collections/recoverytest/state.json] for collection [recoverytest] has= occurred - updating... (live nodes size: [2]) [junit4] 2> 2764265 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= ] o.a.s.s.HttpSolrCall [admin] webapp=3Dnull path=3D/admin/collections = params=3D{replicationFactor=3D2&collection.configName=3Dconf&maxShardsPerNo= de=3D1&name=3Drecoverytest&action=3DCREATE&numShards=3D1&wt=3Djavabin&versi= on=3D2} status=3D0 QTime=3D3610 [junit4] 2> 2764267 INFO (TEST-HdfsRecoveryZkTest.test-seed#[3FFC99BF= 5C01CF22]) [ ] o.a.s.c.RecoveryZkTest Indexing 1200 documents [junit4] 2> 2764295 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-0 (1549805942865920000)]} 0 13 [junit4] 2> 2764295 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-0 (154980594= 2865920000)]} 0 24 [junit4] 2> 2764296 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-0 (1549805942864871424)]} 0 14 [junit4] 2> 2764296 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-0 (154980594= 2864871424)]} 0 25 [junit4] 2> 2764303 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-0 (-1549805942895280128)]} 0 1 [junit4] 2> 2764303 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-0 (-1549805942895280129)]} 0 1 [junit4] 2> 2764303 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-0 (-15498= 05942895280128)]} 0 4 [junit4] 2> 2764303 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-0 (-15498= 05942895280129)]} 0 4 [junit4] 2> 2764308 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-1 (1549805942901571585)]} 0 1 [junit4] 2> 2764308 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-1 (154980594= 2901571585)]} 0 3 [junit4] 2> 2764309 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-1 (1549805942901571584)]} 0 1 [junit4] 2> 2764309 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-1 (154980594= 2901571584)]} 0 3 [junit4] 2> 2764314 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-2 (1549805942907863041)]} 0 1 [junit4] 2> 2764314 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-2 (1549805942907863040)]} 0 1 [junit4] 2> 2764314 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-2 (154980594= 2907863040)]} 0 3 [junit4] 2> 2764314 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-2 (154980594= 2907863041)]} 0 3 [junit4] 2> 2764319 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-2 (-1549805942913105920)]} 0 1 [junit4] 2> 2764319 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-2 (-1549805942913105921)]} 0 1 [junit4] 2> 2764319 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-2 (-15498= 05942913105920)]} 0 3 [junit4] 2> 2764320 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-2 (-15498= 05942913105921)]} 0 3 [junit4] 2> 2764325 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-3 (1549805942919397377)]} 0 0 [junit4] 2> 2764325 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-3 (1549805942919397376)]} 0 1 [junit4] 2> 2764326 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-3 (154980594= 2919397377)]} 0 3 [junit4] 2> 2764326 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-3 (154980594= 2919397376)]} 0 3 [junit4] 2> 2764330 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-4 (1549805942925688832)]} 0 1 [junit4] 2> 2764330 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-4 (1549805942925688833)]} 0 0 [junit4] 2> 2764331 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-4 (154980594= 2925688832)]} 0 3 [junit4] 2> 2764331 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-4 (154980594= 2925688833)]} 0 3 [junit4] 2> 2764335 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-5 (1549805942929883136)]} 0 1 [junit4] 2> 2764336 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-5 (1549805942930931712)]} 0 1 [junit4] 2> 2764336 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-5 (154980594= 2929883136)]} 0 3 [junit4] 2> 2764336 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-5 (154980594= 2930931712)]} 0 3 [junit4] 2> 2764343 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-5 (-1549805942936174593)]} 0 1 [junit4] 2> 2764343 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-5 (-1549805942936174592)]} 0 3 [junit4] 2> 2764344 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-5 (-15498= 05942936174593)]} 0 5 [junit4] 2> 2764344 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-5 (-15498= 05942936174592)]} 0 5 [junit4] 2> 2764349 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-6 (1549805942944563201)]} 0 0 [junit4] 2> 2764349 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-6 (1549805942944563200)]} 0 0 [junit4] 2> 2764349 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-6 (154980594= 2944563201)]} 0 2 [junit4] 2> 2764349 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-6 (154980594= 2944563200)]} 0 3 [junit4] 2> 2764357 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-7 (1549805942954000385)]} 0 0 [junit4] 2> 2764357 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-7 (1549805942954000384)]} 0 1 [junit4] 2> 2764358 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-7 (154980594= 2954000385)]} 0 2 [junit4] 2> 2764358 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-7 (154980594= 2954000384)]} 0 3 [junit4] 2> 2764363 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-8 (1549805942959243265)]} 0 1 [junit4] 2> 2764363 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-8 (1549805942959243264)]} 0 1 [junit4] 2> 2764363 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-8 (154980594= 2959243265)]} 0 3 [junit4] 2> 2764363 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-8 (154980594= 2959243264)]} 0 3 [junit4] 2> 2764368 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-6 (-1549805942964486145)]} 0 0 [junit4] 2> 2764368 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-6 (-1549805942964486144)]} 0 1 [junit4] 2> 2764368 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-6 (-15498= 05942964486145)]} 0 2 [junit4] 2> 2764369 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-6 (-15498= 05942964486144)]} 0 3 [junit4] 2> 2764373 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-9 (1549805942970777600)]} 0 1 [junit4] 2> 2764374 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-9 (1549805942969729024)]} 0 1 [junit4] 2> 2764374 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-9 (154980594= 2970777600)]} 0 3 [junit4] 2> 2764374 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-9 (154980594= 2969729024)]} 0 4 [junit4] 2> 2764380 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-10 (1549805942977069057)]} 0 1 [junit4] 2> 2764380 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-10 (1549805942977069056)]} 0 1 [junit4] 2> 2764380 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-10 (15498059= 42977069057)]} 0 3 [junit4] 2> 2764380 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-10 (15498059= 42977069056)]} 0 3 [junit4] 2> 2764385 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-11 (1549805942982311937)]} 0 0 [junit4] 2> 2764385 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-11 (15498059= 42982311937)]} 0 2 [junit4] 2> 2764385 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-11 (1549805942982311936)]} 0 1 [junit4] 2> 2764386 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-11 (15498059= 42982311936)]} 0 4 [junit4] 2> 2764390 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-9 (-1549805942988603392)]} 0 1 [junit4] 2> 2764391 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-9 (-15498= 05942988603392)]} 0 3 [junit4] 2> 2764391 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-9 (-1549805942988603393)]} 0 1 [junit4] 2> 2764391 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-9 (-15498= 05942988603393)]} 0 3 [junit4] 2> 2764396 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-12 (1549805942993846272)]} 0 1 [junit4] 2> 2764396 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-12 (1549805942993846273)]} 0 1 [junit4] 2> 2764396 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-12 (15498059= 42993846272)]} 0 3 [junit4] 2> 2764397 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-12 (15498059= 42993846273)]} 0 3 [junit4] 2> 2764402 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-13 (1549805942999089152)]} 0 1 [junit4] 2> 2764402 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-13 (1549805942999089153)]} 0 2 [junit4] 2> 2764402 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-13 (15498059= 42999089153)]} 0 3 [junit4] 2> 2764402 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-13 (15498059= 42999089152)]} 0 4 [junit4] 2> 2764408 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-11 (-1549805943006429184)]} 0 0 [junit4] 2> 2764408 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-11 (-1549805943005380608)]} 0 1 [junit4] 2> 2764408 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-11 (-1549= 805943006429184)]} 0 3 [junit4] 2> 2764409 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-11 (-1549= 805943005380608)]} 0 4 [junit4] 2> 2764413 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-14 (1549805943011672065)]} 0 0 [junit4] 2> 2764413 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-14 (1549805943011672064)]} 0 1 [junit4] 2> 2764413 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-14 (15498059= 43011672065)]} 0 2 [junit4] 2> 2764413 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-14 (15498059= 43011672064)]} 0 3 [junit4] 2> 2764419 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-15 (1549805943016914944)]} 0 1 [junit4] 2> 2764419 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-15 (1549805943016914945)]} 0 1 [junit4] 2> 2764419 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-15 (15498059= 43016914945)]} 0 3 [junit4] 2> 2764419 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-15 (15498059= 43016914944)]} 0 3 [junit4] 2> 2764424 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-16 (1549805943023206400)]} 0 1 [junit4] 2> 2764424 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-16 (1549805943023206401)]} 0 1 [junit4] 2> 2764424 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-16 (15498059= 43023206400)]} 0 3 [junit4] 2> 2764424 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-16 (15498059= 43023206401)]} 0 3 [junit4] 2> 2764429 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-13 (-1549805943028449280)]} 0 1 [junit4] 2> 2764429 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-13 (-1549805943028449281)]} 0 1 [junit4] 2> 2764429 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-13 (-1549= 805943028449280)]} 0 3 [junit4] 2> 2764429 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-13 (-1549= 805943028449281)]} 0 3 [junit4] 2> 2764434 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-17 (1549805943033692160)]} 0 1 [junit4] 2> 2764434 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-17 (1549805943033692161)]} 0 1 [junit4] 2> 2764434 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-17 (15498059= 43033692160)]} 0 3 [junit4] 2> 2764434 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-17 (15498059= 43033692161)]} 0 3 [junit4] 2> 2764439 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-18 (1549805943038935040)]} 0 1 [junit4] 2> 2764439 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-18 (1549805943038935041)]} 0 1 [junit4] 2> 2764439 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-18 (15498059= 43038935040)]} 0 3 [junit4] 2> 2764440 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-18 (15498059= 43038935041)]} 0 3 [junit4] 2> 2764449 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-18 (-1549805943050469376)]} 0 1 [junit4] 2> 2764450 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-18 (-1549805943050469377)]} 0 1 [junit4] 2> 2764450 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-18 (-1549= 805943050469376)]} 0 3 [junit4] 2> 2764450 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-18 (-1549= 805943050469377)]} 0 3 [junit4] 2> 2764458 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-19 (1549805943058857984)]} 0 1 [junit4] 2> 2764458 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-19 (1549805943058857985)]} 0 1 [junit4] 2> 2764459 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-19 (15498059= 43058857984)]} 0 3 [junit4] 2> 2764459 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-19 (15498059= 43058857985)]} 0 3 [junit4] 2> 2764463 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-20 (1549805943065149441)]} 0 0 [junit4] 2> 2764463 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-20 (1549805943065149440)]} 0 1 [junit4] 2> 2764463 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-20 (15498059= 43065149441)]} 0 2 [junit4] 2> 2764464 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-20 (15498059= 43065149440)]} 0 2 [junit4] 2> 2764468 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-21 (1549805943069343744)]} 0 1 [junit4] 2> 2764469 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-21 (1549805943069343745)]} 0 1 [junit4] 2> 2764469 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-21 (15498059= 43069343744)]} 0 3 [junit4] 2> 2764469 INFO (qtp647022897-27083) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-21 (15498059= 43069343745)]} 0 3 [junit4] 2> 2764474 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-22 (1549805943075635201)]} 0 1 [junit4] 2> 2764474 INFO (qtp1166891557-27094) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-22 (1549805943075635200)]} 0 1 [junit4] 2> 2764474 INFO (qtp647022897-27091) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-22 (15498059= 43075635201)]} 0 3 [junit4] 2> 2764474 INFO (qtp647022897-27080) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-22 (15498059= 43075635200)]} 0 3 [junit4] 2> 2764480 INFO (qtp1166891557-27084) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[2-22 (-1549805943080878081)]} 0 0 [junit4] 2> 2764480 INFO (qtp1166891557-27088) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{delete=3D[1-22 (-1549805943080878080)]} 0 2 [junit4] 2> 2764480 INFO (qtp647022897-27085) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[2-22 (-1549= 805943080878081)]} 0 3 [junit4] 2> 2764480 INFO (qtp647022897-27087) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{delete=3D[1-22 (-1549= 805943080878080)]} 0 3 [junit4] 2> 2764484 INFO (qtp1166891557-27079) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-23 (1549805943087169537)]} 0 1 [junit4] 2> 2764484 INFO (qtp1166891557-27086) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[2-23 (1549805943087169536)]} 0 0 [junit4] 2> 2764485 INFO (qtp647022897-27089) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[1-23 (15498059= 43087169537)]} 0 2 [junit4] 2> 2764485 INFO (qtp647022897-27093) [n:127.0.0.1:44741_solr= c:recoverytest s:shard1 r:core_node2 x:recoverytest_shard1_replica1] o.a.s= .u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica1] webapp=3D/so= lr path=3D/update params=3D{wt=3Djavabin&version=3D2}{add=3D[2-23 (15498059= 43087169536)]} 0 3 [junit4] 2> 2764489 INFO (qtp1166891557-27090) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0.0.1:44741/solr/recoverytest_shard1_replica1/&wt=3Djavabin&version= =3D2}{add=3D[1-24 (1549805943091363840)]} 0 1 [junit4] 2> 2764489 INFO (qtp1166891557-27092) [n:127.0.0.1:55868_sol= r c:recoverytest s:shard1 r:core_node1 x:recoverytest_shard1_replica2] o.a.= s.u.p.LogUpdateProcessorFactory [recoverytest_shard1_replica2] webapp=3D/s= olr path=3D/update params=3D{update.distrib=3DFROMLEADER&distrib.from=3Dhtt= p://127.0 [...truncated too long message...] xTest, TestAuthorizationFramework, RequestHandlersTest, TestDistribDocBased= Version, CollectionsAPISolrJTest, TestSolrCLIRunExample, TestSchemaNameReso= urce, DistanceFunctionTest, TestConfigSets, TestSubQueryTransformerDistrib,= QueryParsingTest, TestDocSet, TestAtomicUpdateErrorCases, TestFieldCacheSo= rt, TestJmxMonitoredMap, TestReversedWildcardFilterFactory, TestLeaderElect= ionWithEmptyReplica, TestDistributedGrouping, TestSolrDeletionPolicy2, Hdfs= DirectoryFactoryTest, DeleteInactiveReplicaTest, DistributedFacetPivotLarge= Test, ShowFileRequestHandlerTest, DistribCursorPagingTest, TestQueryWrapper= Filter, DistribJoinFromCollectionTest, TestOnReconnectListenerSupport, Test= SearchPerf, CollectionReloadTest, HighlighterConfigTest, DeleteShardTest, T= estFileDictionaryLookup, SpellCheckCollatorTest, ReturnFieldsTest, SolrCLIZ= kUtilsTest, TermVectorComponentTest, TestLockTree, AnalyticsMergeStrategyTe= st, TestQuerySenderNoQuery, IgnoreCommitOptimizeUpdateProcessorFactoryTest,= URLClassifyProcessorTest, SparseHLLTest, AtomicUpdatesTest, TestLMDirichle= tSimilarityFactory, TestLegacyFieldCache, HdfsChaosMonkeySafeLeaderTest, Te= stGraphMLResponseWriter, CollectionTooManyReplicasTest, SuggestComponentCon= textFilterQueryTest, TestReload, TestUseDocValuesAsStored, CdcrVersionRepli= cationTest, BlockDirectoryTest, TestSlowCompositeReaderWrapper, TestConfig,= DefaultValueUpdateProcessorTest, TestJsonFacetRefinement, TestValueSourceC= ache, TestSolrConfigHandlerCloud, BlobRepositoryCloudTest, UpdateRequestPro= cessorFactoryTest, CursorMarkTest, TestObjectReleaseTracker, BlockJoinFacet= DistribTest, SuggesterTSTTest, SolrInfoMBeanTest, ClusterStateTest, TestDis= tributedStatsComponentCardinality, TestReplicaProperties, SliceStateTest, T= estLeaderInitiatedRecoveryThread, TestDistributedMissingSort, TestQuerySend= erListener, TestSolr4Spatial, SSLMigrationTest, RuleEngineTest, SearchHandl= erTest, ReplaceNodeTest, TestFuzzyAnalyzedSuggestions, FieldAnalysisRequest= HandlerTest, OverseerCollectionConfigSetProcessorTest, CleanupOldIndexTest,= TestSubQueryTransformer, TestCloudManagedSchema, PeerSyncWithIndexFingerpr= intCachingTest, ScriptEngineTest, TestCollapseQParserPlugin, IndexSchemaTes= t, TestHashQParserPlugin, SecurityConfHandlerTest, DateMathParserTest, Solr= IndexSplitterTest, TestMacros, TestFieldTypeCollectionResource, TestIndexSe= archer, JavabinLoaderTest, TestManagedSchema, TestJoin, DistributedInterval= FacetingTest, TestBackupRepositoryFactory, TestSmileRequest, TestInitParams= , LeaderElectionIntegrationTest, TestIndexingPerformance, TestSystemIdResol= ver, MBeansHandlerTest, TestRequestStatusCollectionAPI, SharedFSAutoReplica= FailoverUtilsTest, PreAnalyzedUpdateProcessorTest, ExternalFileFieldSortTes= t, TestSolrConfigHandler, TestSimpleTrackingShardHandler, TestAnalyzeInfixS= uggestions, DistributedTermsComponentTest, LukeRequestHandlerTest, TestExac= tStatsCache, DistributedVersionInfoTest, MinimalSchemaTest, WordBreakSolrSp= ellCheckerTest, LeaderFailoverAfterPartitionTest, TestOmitPositions, TestSo= lrFieldCacheMBean, TestIntervalFaceting, CreateCollectionCleanupTest, Field= MutatingUpdateProcessorTest, TestNumericTerms64, TestCoreDiscovery, BasicZk= Test, TestWriterPerf, PreAnalyzedFieldTest, TestSolrQueryParserResource, HL= LSerializationTest, ZkStateWriterTest, SpellingQueryConverterTest, TestMana= gedStopFilterFactory, JsonLoaderTest, TestSchemaSimilarityResource, ChaosMo= nkeySafeLeaderTest, TestPHPSerializedResponseWriter, HttpPartitionTest, Dis= tributedQueryComponentOptimizationTest, DirectoryFactoryTest, SmileWriterTe= st, SpatialRPTFieldTypeTest, TestFieldCacheVsDocValues, TestStressUserVersi= ons, SuggesterWFSTTest, CopyFieldTest, BasicDistributedZkTest, BasicDistrib= utedZk2Test, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreS= tressTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandom= Faceting, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, ZkCLITest, Sh= ardRoutingCustomTest, AutoCommitTest, TestFaceting, TestRecovery, TestHashP= artitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistribu= tedTest, TestRealTimeGet, TestStressReorder, HardAutoCommitTest, TestCoreCo= ntainer, StatsComponentTest, SpellCheckComponentTest, TestGroupingSearch, S= olrCmdDistributorTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTes= t, TestSort, TestFiltering, TestFunctionQuery, BasicFunctionalityTest, Dire= ctUpdateHandlerTest, TestBadConfig, HighlighterTest, CurrencyFieldOpenExcha= ngeTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, = AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, TestExten= dedDismaxParser, SuggesterFSTTest, CoreAdminHandlerTest, SuggesterTest, Tes= tStressLucene, SpatialFilterTest, NoCacheHeaderTest, SchemaVersionSpecificB= ehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, Direct= UpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, SortB= yFunctionTest, TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPol= icy1, DebugComponentTest, CacheHeaderTest, DisMaxRequestHandlerTest, TestQu= eryUtils, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, T= estQueryTypes, FileBasedSpellCheckerTest, MoreLikeThisHandlerTest, TestSolr= QueryParser, LoggingHandlerTest, TestCSVResponseWriter, BinaryUpdateRequest= HandlerTest, CSVRequestHandlerTest, TestComponentsName, TestBinaryResponseW= riter, TestLFUCache, SOLR749Test, UpdateParamsTest, SolrIndexConfigTest, Ba= dComponentTest, TestStressRecovery, TestMergePolicyConfig, MultiTermTest, S= ampleTest, TestBinaryField, TestSolrCoreProperties, TestPostingsSolrHighlig= hter, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, SpellPossibilityIte= ratorTest, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestSwee= tSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, T= estLMJelinekMercerSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, Res= ourceLoaderTest, TestFastOutputStream, TestFastLRUCache, TestSuggestSpellin= gConverter, DateFieldTest, RAMDirectoryFactoryTest, TestSolrJ, TestLRUCache= , TestUtils, TestDocumentBuilder, SystemInfoHandlerTest, UUIDFieldTest, Fil= eUtilsTest, CircularListTest, DistributedMLTComponentTest, TestRTGBase, Cur= sorPagingTest, TestCrossCoreJoin, TestHighlightDedupGrouping, TestTolerantS= earch, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ConnectionR= euseTest, ActionThrottleTest, AliasIntegrationTest, AsyncCallRequestStatusR= esponseTest, CdcrBootstrapTest, ConcurrentDeleteAndCreateCollectionTest, Co= nfigSetsAPITest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, Delete= ReplicaTest, DistributedQueueTest, ForceLeaderTest, OverseerStatusTest, Ove= rseerTaskQueueTest, RestartWhileUpdatingTest, RollingRestartTest, SimpleCol= lectionCreateDeleteTest, SolrXmlInZkTest, TestCloudDeleteByQuery, TestCloud= InspectUtil, TestCloudPseudoReturnFields, TestCloudRecovery, TestClusterPro= perties, TestCollectionAPI, TestConfigSetsAPI, TestConfigSetsAPIExclusivity= , TestConfigSetsAPIZkFailure, TestCryptoKeys, TestDeleteCollectionOnDownNod= es, TestExclusionRuleCollectionAccess, TestLeaderElectionZkExpiry, TestMini= SolrCloudCluster, TestMiniSolrCloudClusterSSL, TestRandomRequestDistributio= n, TestSolrCloudWithDelegationTokens, TestStressCloudBlindAtomicUpdates, Tl= ogReplayBufferedWhileIndexingTest, VMParamsZkACLAndCredentialsProvidersTest= , HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsCollectionsA= PIDistributedZkTest, HdfsNNFailoverTest, HdfsRecoveryZkTest] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=3DHdfsRecoveryZ= kTest -Dtests.seed=3D3FFC99BF5C01CF22 -Dtests.slow=3Dtrue -Dtests.locale=3D= nn-NO -Dtests.timezone=3DAmerica/Creston -Dtests.asserts=3Dtrue -Dtests.fil= e.encoding=3DUTF-8 [junit4] ERROR 0.00s J1 | HdfsRecoveryZkTest (suite) <<< [junit4] > Throwable #1: java.lang.AssertionError: ObjectTracker foun= d 1 object(s) that were not released!!! [HdfsTransactionLog] [junit4] > org.apache.solr.common.util.ObjectReleaseTracker$ObjectTra= ckerException [junit4] > =09at org.apache.solr.common.util.ObjectReleaseTracker.tra= ck(ObjectReleaseTracker.java:43) [junit4] > =09at org.apache.solr.update.HdfsTransactionLog.(Hdf= sTransactionLog.java:130) [junit4] > =09at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdate= Log.java:202) [junit4] > =09at org.apache.solr.update.UpdateHandler.(UpdateHa= ndler.java:137) [junit4] > =09at org.apache.solr.update.UpdateHandler.(UpdateHa= ndler.java:94) [junit4] > =09at org.apache.solr.update.DirectUpdateHandler2.(D= irectUpdateHandler2.java:102) [junit4] > =09at java.lang.reflect.Constructor.newInstance(Constructo= r.java:423) [junit4] > =09at org.apache.solr.core.SolrCore.createInstance(SolrCor= e.java:706) [junit4] > =09at org.apache.solr.core.SolrCore.createUpdateHandler(So= lrCore.java:768) [junit4] > =09at org.apache.solr.core.SolrCore.initUpdateHandler(Solr= Core.java:1007) [junit4] > =09at org.apache.solr.core.SolrCore.(SolrCore.java:8= 72) [junit4] > =09at org.apache.solr.core.SolrCore.(SolrCore.java:7= 76) [junit4] > =09at org.apache.solr.core.CoreContainer.create(CoreContai= ner.java:842) [junit4] > =09at org.apache.solr.core.CoreContainer.lambda$load$0(Cor= eContainer.java:498) [junit4] > =09at java.util.concurrent.FutureTask.run(FutureTask.java:= 266) [junit4] > =09at org.apache.solr.common.util.ExecutorUtil$MDCAwareThr= eadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) [junit4] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(Th= readPoolExecutor.java:1142) [junit4] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(T= hreadPoolExecutor.java:617) [junit4] > =09at java.lang.Thread.run(Thread.java:745) [junit4] > =09at __randomizedtesting.SeedInfo.seed([3FFC99BF5C01CF22]= :0) [junit4] > =09at org.apache.solr.SolrTestCaseJ4.teardownTestCases(Sol= rTestCaseJ4.java:260) [junit4] > =09at java.lang.Thread.run(Thread.java:745) [junit4] Completed [525/648 (1!)] on J1 in 53.11s, 1 test, 1 failure <<<= FAILURES! [...truncated 60811 lines...] ------=_Part_178_309502453.1478011636011 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org ------=_Part_178_309502453.1478011636011--