lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-6.x - Build # 601 - Still Unstable
Date Thu, 15 Dec 2016 15:31:20 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/601/

1 tests failed.
FAILED:  org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=3086, name=SocketProxy-Response-58341:44485, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3086, name=SocketProxy-Response-58341:44485, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
	at __randomizedtesting.SeedInfo.seed([978136A83E311B2D:1FD5097290CD76D5]:0)
Caused by: java.lang.RuntimeException: java.net.SocketException: Socket is closed
	at __randomizedtesting.SeedInfo.seed([978136A83E311B2D]:0)
	at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:347)
Caused by: java.net.SocketException: Socket is closed
	at java.net.Socket.setSoTimeout(Socket.java:1137)
	at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:344)




Build Log:
[...truncated 10995 lines...]
   [junit4] Suite: org.apache.solr.cloud.LeaderInitiatedRecoveryOnCommitTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/init-core-data-001
   [junit4]   2> 275131 INFO  (SUITE-LeaderInitiatedRecoveryOnCommitTest-seed#[978136A83E311B2D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 275131 INFO  (SUITE-LeaderInitiatedRecoveryOnCommitTest-seed#[978136A83E311B2D]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 275150 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 275158 INFO  (Thread-643) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 275158 INFO  (Thread-643) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 275257 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkTestServer start zk server on port:46421
   [junit4]   2> 275531 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 275534 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 275537 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 275538 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 275540 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 275548 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 275562 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 275564 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 275565 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 275566 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 275568 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 275996 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/control-001/cores/collection1
   [junit4]   2> 276006 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 276023 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6aeedd9e{/,null,AVAILABLE}
   [junit4]   2> 276023 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5f26d6ec{HTTP/1.1,[http/1.1]}{127.0.0.1:45614}
   [junit4]   2> 276023 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server Started @280519ms
   [junit4]   2> 276023 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/tempDir-001/control/data, hostContext=/, hostPort=60734, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/control-001/cores}
   [junit4]   2> 276032 ERROR (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 276032 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.4.0
   [junit4]   2> 276032 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 276032 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 276032 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-15T14:39:55.167Z
   [junit4]   2> 276050 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 276050 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/control-001/solr.xml
   [junit4]   2> 276062 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 276070 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46421/solr
   [junit4]   2> 276137 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:60734_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60734_
   [junit4]   2> 276138 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:60734_    ] o.a.s.c.Overseer Overseer (id=97112083287441412-127.0.0.1:60734_-n_0000000000) starting
   [junit4]   2> 276173 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:60734_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60734_
   [junit4]   2> 276176 INFO  (zkCallback-540-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 276287 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:60734_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/control-001/cores
   [junit4]   2> 276287 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:60734_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 276327 INFO  (OverseerStateUpdate-97112083287441412-127.0.0.1:60734_-n_0000000000) [n:127.0.0.1:60734_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 277511 WARN  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 277512 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 277535 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 277757 WARN  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 277793 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 277830 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 277831 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/control-001/cores/collection1/data/]
   [junit4]   2> 277831 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e9aa36b
   [junit4]   2> 277842 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7311198731128292]
   [junit4]   2> 278114 WARN  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 278135 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 278135 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 278136 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 278136 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 278136 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=72.625, floorSegmentMB=1.6845703125, forceMergeDeletesPctAllowed=4.546658267128728, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 278137 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@370b816b[collection1] main]
   [junit4]   2> 278138 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 278138 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 278138 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 278139 INFO  (coreLoadExecutor-1572-thread-1-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1553793335714381824
   [junit4]   2> 278141 INFO  (searcherExecutor-1573-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@370b816b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 278162 INFO  (coreZkRegister-1565-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 278162 INFO  (coreZkRegister-1565-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 278162 INFO  (coreZkRegister-1565-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60734/collection1/
   [junit4]   2> 278162 INFO  (coreZkRegister-1565-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 278162 INFO  (coreZkRegister-1565-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:60734/collection1/ has no replicas
   [junit4]   2> 278167 INFO  (coreZkRegister-1565-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60734/collection1/ shard1
   [junit4]   2> 278318 INFO  (coreZkRegister-1565-thread-1-processing-n:127.0.0.1:60734_ x:collection1 c:control_collection) [n:127.0.0.1:60734_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 278370 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 278371 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:46421/solr ready
   [junit4]   2> 278371 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 278372 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 278529 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-1-001/cores/collection1
   [junit4]   2> 278530 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-1-001
   [junit4]   2> 278535 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 278599 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2b24d256{/,null,AVAILABLE}
   [junit4]   2> 278599 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@37e8084d{HTTP/1.1,[http/1.1]}{127.0.0.1:58341}
   [junit4]   2> 278599 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server Started @283094ms
   [junit4]   2> 278599 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=40876, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-1-001/cores}
   [junit4]   2> 278600 ERROR (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 278600 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.4.0
   [junit4]   2> 278600 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 278600 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 278600 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-15T14:39:57.735Z
   [junit4]   2> 278665 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 278665 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-1-001/solr.xml
   [junit4]   2> 278691 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 278695 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46421/solr
   [junit4]   2> 278744 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 278763 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:40876_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40876_
   [junit4]   2> 278765 INFO  (zkCallback-544-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 278765 INFO  (zkCallback-540-thread-2-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 278765 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 279196 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:40876_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-1-001/cores
   [junit4]   2> 279196 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:40876_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 279210 INFO  (OverseerStateUpdate-97112083287441412-127.0.0.1:60734_-n_0000000000) [n:127.0.0.1:60734_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 279313 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 280225 WARN  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 280226 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 280270 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 280556 WARN  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 280559 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 280586 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 280587 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-1-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 280588 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e9aa36b
   [junit4]   2> 280589 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7311198731128292]
   [junit4]   2> 280594 WARN  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 280646 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 280646 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 280647 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 280647 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 280650 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=72.625, floorSegmentMB=1.6845703125, forceMergeDeletesPctAllowed=4.546658267128728, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 280651 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@59cca3dc[collection1] main]
   [junit4]   2> 280653 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 280653 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 280653 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 280656 INFO  (coreLoadExecutor-1583-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1553793338353647616
   [junit4]   2> 280679 INFO  (searcherExecutor-1584-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@59cca3dc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 280706 INFO  (coreZkRegister-1578-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 280706 INFO  (coreZkRegister-1578-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 280706 INFO  (coreZkRegister-1578-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40876/collection1/
   [junit4]   2> 280706 INFO  (coreZkRegister-1578-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 280706 INFO  (coreZkRegister-1578-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40876/collection1/ has no replicas
   [junit4]   2> 280711 INFO  (coreZkRegister-1578-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40876/collection1/ shard1
   [junit4]   2> 280814 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 280862 INFO  (coreZkRegister-1578-thread-1-processing-n:127.0.0.1:40876_ x:collection1 c:collection1) [n:127.0.0.1:40876_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 280980 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 281060 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-2-001/cores/collection1
   [junit4]   2> 281062 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-2-001
   [junit4]   2> 281063 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 281066 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e427f71{/,null,AVAILABLE}
   [junit4]   2> 281066 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@22a7d824{HTTP/1.1,[http/1.1]}{127.0.0.1:50096}
   [junit4]   2> 281066 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server Started @285561ms
   [junit4]   2> 281066 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=37195, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-2-001/cores}
   [junit4]   2> 281066 ERROR (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 281067 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.4.0
   [junit4]   2> 281067 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 281067 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 281067 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-15T14:40:00.202Z
   [junit4]   2> 281095 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 281095 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-2-001/solr.xml
   [junit4]   2> 281113 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 281122 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46421/solr
   [junit4]   2> 281228 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 281233 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:37195_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37195_
   [junit4]   2> 281234 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281234 INFO  (zkCallback-544-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281235 INFO  (zkCallback-540-thread-5-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281243 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 281351 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 281464 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:37195_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-2-001/cores
   [junit4]   2> 281464 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:37195_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 281471 INFO  (OverseerStateUpdate-97112083287441412-127.0.0.1:60734_-n_0000000000) [n:127.0.0.1:60734_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 281579 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 281579 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 282535 WARN  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 282536 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 282567 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 282712 WARN  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 282726 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 282760 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 282760 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-2-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-2-001/cores/collection1/data/]
   [junit4]   2> 282762 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e9aa36b
   [junit4]   2> 282778 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7311198731128292]
   [junit4]   2> 282806 WARN  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 282854 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 282854 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 282855 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 282855 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 282858 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=72.625, floorSegmentMB=1.6845703125, forceMergeDeletesPctAllowed=4.546658267128728, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 282858 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1b7baf3b[collection1] main]
   [junit4]   2> 282860 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 282861 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 282861 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 282861 INFO  (coreLoadExecutor-1594-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1553793340665757696
   [junit4]   2> 282864 INFO  (searcherExecutor-1595-thread-1-processing-n:127.0.0.1:37195_ x:collection1 c:collection1) [n:127.0.0.1:37195_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1b7baf3b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 282876 INFO  (coreZkRegister-1589-thread-1-processing-n:127.0.0.1:37195_ x:collection1 c:collection1) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 282890 INFO  (updateExecutor-553-thread-1-processing-n:127.0.0.1:37195_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 282973 INFO  (recoveryExecutor-554-thread-1-processing-n:127.0.0.1:37195_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 282974 INFO  (recoveryExecutor-554-thread-1-processing-n:127.0.0.1:37195_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 282974 INFO  (recoveryExecutor-554-thread-1-processing-n:127.0.0.1:37195_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 282974 INFO  (recoveryExecutor-554-thread-1-processing-n:127.0.0.1:37195_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 282974 INFO  (recoveryExecutor-554-thread-1-processing-n:127.0.0.1:37195_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:40876/collection1/] and I am [http://127.0.0.1:37195/collection1/]
   [junit4]   2> 282985 INFO  (recoveryExecutor-554-thread-1-processing-n:127.0.0.1:37195_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:37195_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:40876]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:37195_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 282988 INFO  (SocketProxy-Acceptor-40876) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=44432,localport=40876], receiveBufferSize:531000
   [junit4]   2> 283014 INFO  (SocketProxy-Acceptor-40876) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58341,localport=36777], receiveBufferSize=530904
   [junit4]   2> 283026 INFO  (qtp75889157-2922) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 283026 INFO  (qtp75889157-2922) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 283026 INFO  (qtp75889157-2922) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:37195_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:37195","node_name":"127.0.0.1:37195_","state":"down"}
   [junit4]   2> 283080 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 283080 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 283314 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-3-001/cores/collection1
   [junit4]   2> 283316 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-3-001
   [junit4]   2> 283317 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 283543 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ee6edf0{/,null,AVAILABLE}
   [junit4]   2> 283544 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@5d988255{HTTP/1.1,[http/1.1]}{127.0.0.1:39512}
   [junit4]   2> 283544 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server Started @288039ms
   [junit4]   2> 283544 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/tempDir-001/jetty3, solrconfig=solrconfig.xml, hostContext=/, hostPort=56938, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-3-001/cores}
   [junit4]   2> 283544 ERROR (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 283544 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.4.0
   [junit4]   2> 283544 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 283544 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 283545 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-15T14:40:02.680Z
   [junit4]   2> 283575 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 283575 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-3-001/solr.xml
   [junit4]   2> 283588 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 283589 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46421/solr
   [junit4]   2> 283628 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 283632 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:56938_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56938_
   [junit4]   2> 283634 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 283634 INFO  (zkCallback-540-thread-5-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 283638 INFO  (zkCallback-544-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 283638 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 283658 INFO  (zkCallback-563-thread-2-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 283678 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:56938_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-3-001/cores
   [junit4]   2> 283678 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:56938_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 283688 INFO  (OverseerStateUpdate-97112083287441412-127.0.0.1:60734_-n_0000000000) [n:127.0.0.1:60734_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 283792 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 283792 INFO  (zkCallback-563-thread-2-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 283792 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 284027 INFO  (qtp75889157-2922) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:37195_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:37195","node_name":"127.0.0.1:37195_","state":"recovering"}
   [junit4]   2> 284027 INFO  (qtp75889157-2922) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 284027 INFO  (qtp75889157-2922) [n:127.0.0.1:40876_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:37195_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
   [junit4]   2> 284715 WARN  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 284716 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 284755 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 285047 WARN  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 285050 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 285064 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 285065 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-3-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-3-001/cores/collection1/data/]
   [junit4]   2> 285065 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e9aa36b
   [junit4]   2> 285066 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7311198731128292]
   [junit4]   2> 285106 WARN  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 285123 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 285123 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 285123 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 285123 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 285128 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=72.625, floorSegmentMB=1.6845703125, forceMergeDeletesPctAllowed=4.546658267128728, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 285138 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@253bf7b7[collection1] main]
   [junit4]   2> 285141 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 285142 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 285142 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 285143 INFO  (searcherExecutor-1606-thread-1-processing-n:127.0.0.1:56938_ x:collection1 c:collection1) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@253bf7b7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 285143 INFO  (coreLoadExecutor-1605-thread-1-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1553793343058608128
   [junit4]   2> 285159 INFO  (coreZkRegister-1600-thread-1-processing-n:127.0.0.1:56938_ x:collection1 c:collection1) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 285162 INFO  (updateExecutor-560-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 285162 INFO  (recoveryExecutor-561-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 285163 INFO  (recoveryExecutor-561-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 285163 INFO  (recoveryExecutor-561-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 285163 INFO  (recoveryExecutor-561-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 285163 INFO  (recoveryExecutor-561-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:40876/collection1/] and I am [http://127.0.0.1:56938/collection1/]
   [junit4]   2> 285166 INFO  (recoveryExecutor-561-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:40876]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:56938_&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 285168 INFO  (SocketProxy-Acceptor-40876) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=44444,localport=40876], receiveBufferSize:531000
   [junit4]   2> 285169 INFO  (qtp75889157-2921) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 285174 INFO  (qtp75889157-2921) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 285174 INFO  (qtp75889157-2921) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:56938_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:56938","node_name":"127.0.0.1:56938_","state":"down"}
   [junit4]   2> 285185 INFO  (SocketProxy-Acceptor-40876) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58341,localport=36789], receiveBufferSize=530904
   [junit4]   2> 285268 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 285268 INFO  (zkCallback-563-thread-2-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 285269 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 285367 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-4-001/cores/collection1
   [junit4]   2> 285368 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-4-001
   [junit4]   2> 285368 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 285370 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64ca511a{/,null,AVAILABLE}
   [junit4]   2> 285370 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@4a8d233e{HTTP/1.1,[http/1.1]}{127.0.0.1:51149}
   [junit4]   2> 285370 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.Server Started @289866ms
   [junit4]   2> 285370 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/tempDir-001/jetty4, solrconfig=solrconfig.xml, hostContext=/, hostPort=46533, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-4-001/cores}
   [junit4]   2> 285371 ERROR (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 285371 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.4.0
   [junit4]   2> 285371 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 285371 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 285371 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-15T14:40:04.506Z
   [junit4]   2> 285387 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 285387 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-4-001/solr.xml
   [junit4]   2> 285402 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 285403 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46421/solr
   [junit4]   2> 285464 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:46533_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 285470 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:46533_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46533_
   [junit4]   2> 285472 INFO  (zkCallback-544-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 285473 INFO  (zkCallback-563-thread-2-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 285473 INFO  (zkCallback-570-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 285473 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 285473 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 285474 INFO  (zkCallback-540-thread-2-processing-n:127.0.0.1:60734_) [n:127.0.0.1:60734_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
   [junit4]   2> 285558 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:46533_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-4-001/cores
   [junit4]   2> 285559 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [n:127.0.0.1:46533_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 285564 INFO  (OverseerStateUpdate-97112083287441412-127.0.0.1:60734_-n_0000000000) [n:127.0.0.1:60734_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 285667 INFO  (zkCallback-563-thread-2-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 285667 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 285667 INFO  (zkCallback-570-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 285667 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 286174 INFO  (qtp75889157-2921) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:56938_, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"collection1","base_url":"http://127.0.0.1:56938","node_name":"127.0.0.1:56938_","state":"recovering"}
   [junit4]   2> 286174 INFO  (qtp75889157-2921) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 286174 INFO  (qtp75889157-2921) [n:127.0.0.1:40876_    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:56938_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1005
   [junit4]   2> 286606 WARN  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 286607 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 286623 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 286717 WARN  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 286719 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 286749 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 286750 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-4-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderInitiatedRecoveryOnCommitTest_978136A83E311B2D-001/shard-4-001/cores/collection1/data/]
   [junit4]   2> 286750 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5e9aa36b
   [junit4]   2> 286751 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7311198731128292]
   [junit4]   2> 286759 WARN  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 286777 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 286777 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 286778 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 286778 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 286779 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=72.625, floorSegmentMB=1.6845703125, forceMergeDeletesPctAllowed=4.546658267128728, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 286779 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c9e718d[collection1] main]
   [junit4]   2> 286780 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 286780 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 286781 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 286781 INFO  (coreLoadExecutor-1616-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1553793344776175616
   [junit4]   2> 286785 INFO  (searcherExecutor-1617-thread-1-processing-n:127.0.0.1:46533_ x:collection1 c:collection1) [n:127.0.0.1:46533_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@c9e718d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 286787 INFO  (coreZkRegister-1611-thread-1-processing-n:127.0.0.1:46533_ x:collection1 c:collection1) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
   [junit4]   2> 286788 INFO  (updateExecutor-567-thread-1-processing-n:127.0.0.1:46533_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
   [junit4]   2> 286788 INFO  (recoveryExecutor-568-thread-1-processing-n:127.0.0.1:46533_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
   [junit4]   2> 286789 INFO  (recoveryExecutor-568-thread-1-processing-n:127.0.0.1:46533_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
   [junit4]   2> 286789 INFO  (recoveryExecutor-568-thread-1-processing-n:127.0.0.1:46533_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
   [junit4]   2> 286789 INFO  (recoveryExecutor-568-thread-1-processing-n:127.0.0.1:46533_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
   [junit4]   2> 286789 INFO  (recoveryExecutor-568-thread-1-processing-n:127.0.0.1:46533_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:40876/collection1/] and I am [http://127.0.0.1:46533/collection1/]
   [junit4]   2> 286793 INFO  (recoveryExecutor-568-thread-1-processing-n:127.0.0.1:46533_ x:collection1 s:shard1 c:collection1 r:core_node4) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:40876]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:46533_&coreNodeName=core_node4&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
   [junit4]   2> 286795 INFO  (SocketProxy-Acceptor-40876) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=44452,localport=40876], receiveBufferSize:531000
   [junit4]   2> 286802 INFO  (SocketProxy-Acceptor-40876) [    ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=58341,localport=36797], receiveBufferSize=530904
   [junit4]   2> 286803 INFO  (qtp75889157-2923) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 286803 INFO  (qtp75889157-2923) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 286803 INFO  (qtp75889157-2923) [n:127.0.0.1:40876_    ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:46533_, coreNodeName=core_node4, onlyIfActiveCheckResult=false, nodeProps: core_node4:{"core":"collection1","base_url":"http://127.0.0.1:46533","node_name":"127.0.0.1:46533_","state":"down"}
   [junit4]   2> 286894 INFO  (zkCallback-563-thread-2-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 286894 INFO  (zkCallback-556-thread-1-processing-n:127.0.0.1:37195_) [n:127.0.0.1:37195_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 286894 INFO  (zkCallback-550-thread-1-processing-n:127.0.0.1:40876_) [n:127.0.0.1:40876_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 286894 INFO  (zkCallback-570-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [5])
   [junit4]   2> 287082 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 287082 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.LeaderInitiatedRecoveryOnCommitTest Running oneShardTest
   [junit4]   2> 287084 INFO  (SocketProxy-Acceptor-40876) [    ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=44454,localport=40876], receiveBufferSize:531000
   [junit4]   2> 287090 INFO  (qtp75889157-2922) [n:127.0.0.1:40876_    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=3&maxShardsPerNode=1&name=c8n_1x3_commits

[...truncated too long message...]

olrCore org.apache.solr.core.SolrCore@27314396
   [junit4]   2> 310215 WARN  (recoveryExecutor-561-thread-1-processing-n:127.0.0.1:56938_ x:collection1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:56938_ c:collection1 s:shard1 r:core_node3 x:collection1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection1] coreNodeName=[core_node3]
   [junit4]   2> 310221 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.Overseer Overseer (id=97112083287441423-127.0.0.1:56938_-n_0000000003) closing
   [junit4]   2> 310221 INFO  (OverseerStateUpdate-97112083287441423-127.0.0.1:56938_-n_0000000003) [n:127.0.0.1:56938_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:56938_
   [junit4]   2> 310228 INFO  (zkCallback-570-thread-2-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 310228 INFO  (zkCallback-570-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46533_
   [junit4]   2> 310228 INFO  (zkCallback-544-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 310229 INFO  (zkCallback-570-thread-1-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_    ] o.a.s.c.Overseer Overseer (id=97112083287441426-127.0.0.1:46533_-n_0000000004) starting
   [junit4]   2> 311722 WARN  (zkCallback-563-thread-2-processing-n:127.0.0.1:56938_) [n:127.0.0.1:56938_    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 311725 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@5d988255{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 311725 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ee6edf0{/,null,UNAVAILABLE}
   [junit4]   2> 311728 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 46533
   [junit4]   2> 311728 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1606792420
   [junit4]   2> 311737 INFO  (coreCloseExecutor-1673-thread-1) [n:127.0.0.1:46533_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6e369bf4
   [junit4]   2> 311770 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.Overseer Overseer (id=97112083287441426-127.0.0.1:46533_-n_0000000004) closing
   [junit4]   2> 311770 INFO  (OverseerStateUpdate-97112083287441426-127.0.0.1:46533_-n_0000000004) [n:127.0.0.1:46533_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:46533_
   [junit4]   2> 311773 INFO  (zkCallback-544-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 313273 WARN  (zkCallback-570-thread-2-processing-n:127.0.0.1:46533_) [n:127.0.0.1:46533_    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 313280 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@4a8d233e{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 313280 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@64ca511a{/,null,UNAVAILABLE}
   [junit4]   2> 313286 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 60734
   [junit4]   2> 313287 INFO  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46421 46421
   [junit4]   2> 313390 INFO  (Thread-643) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46421 46421
   [junit4]   2> 313398 WARN  (Thread-643) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	5	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/c8n_2x2_commits/state.json
   [junit4]   2> 	4	/solr/collections/collection1/state.json
   [junit4]   2> 	3	/solr/collections/c8n_1x3_commits/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	56	/solr/overseer/queue
   [junit4]   2> 	31	/solr/overseer/collection-queue-work
   [junit4]   2> 	21	/solr/overseer/queue-work
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 
   [junit4]   2> 313398 WARN  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SocketProxy Closing 1 connections to: http://127.0.0.1:56938/, target: http://127.0.0.1:39512/
   [junit4]   2> 313398 WARN  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SocketProxy Closing 3 connections to: http://127.0.0.1:46533/, target: http://127.0.0.1:51149/
   [junit4]   2> 313398 WARN  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SocketProxy Closing 1 connections to: http://127.0.0.1:60734/, target: http://127.0.0.1:45614/
   [junit4]   2> 313398 WARN  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SocketProxy Closing 3 connections to: http://127.0.0.1:37195/, target: http://127.0.0.1:50096/
   [junit4]   2> 313398 WARN  (TEST-LeaderInitiatedRecoveryOnCommitTest.test-seed#[978136A83E311B2D]) [    ] o.a.s.c.SocketProxy Closing 4 connections to: http://127.0.0.1:40876/, target: http://127.0.0.1:58341/
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=LeaderInitiatedRecoveryOnCommitTest -Dtests.method=test -Dtests.seed=978136A83E311B2D -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sq -Dtests.timezone=America/Indianapolis -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   38.3s J1 | LeaderInitiatedRecoveryOnCommitTest.test <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3086, name=SocketProxy-Response-58341:44485, state=RUNNABLE, group=TGRP-LeaderInitiatedRecoveryOnCommitTest]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([978136A83E311B2D:1FD5097290CD76D5]:0)
   [junit4]    > Caused by: java.lang.RuntimeException: java.net.SocketException: Socket is closed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([978136A83E311B2D]:0)
   [junit4]    > 	at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:347)
   [junit4]    > Caused by: java.net.SocketException: Socket is closed
   [junit4]    > 	at java.net.Socket.setSoTimeout(Socket.java:1137)
   [junit4]    > 	at org.apache.solr.cloud.SocketProxy$Bridge$Pump.run(SocketProxy.java:344)
   [junit4]   2> 313402 INFO  (SUITE-LeaderInitiatedRecoveryOnCommitTest-seed#[978136A83E311B2D]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Dec 15, 2016 2:40:32 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=1064, maxMBSortInHeap=5.690530284068884, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=sq, timezone=America/Indianapolis
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=89609384,total=435159040
   [junit4]   2> NOTE: All tests run in this JVM: [TestBackupRepositoryFactory, TestJettySolrRunner, TestWordDelimiterFilterFactory, JsonLoaderTest, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, BaseCdcrDistributedZkTest, TestSystemIdResolver, StatsReloadRaceTest, CollectionReloadTest, TestTrie, TermsComponentTest, DocValuesTest, TestElisionMultitermQuery, TestFieldResource, TestComplexPhraseQParserPlugin, HdfsRestartWhileUpdatingTest, AddSchemaFieldsUpdateProcessorFactoryTest, VMParamsZkACLAndCredentialsProvidersTest, TestNumericTerms32, DirectSolrConnectionTest, ReplicationFactorTest, WordBreakSolrSpellCheckerTest, TestTrackingShardHandlerFactory, FacetPivotSmallTest, SparseHLLTest, TestClusterProperties, TestBM25SimilarityFactory, OverseerCollectionConfigSetProcessorTest, ExitableDirectoryReaderTest, TestAuthenticationFramework, TestRequestForwarding, TestCharFilters, OpenCloseCoreStressTest, TestMacros, TestSegmentSorting, TestManagedSchema, TolerantUpdateProcessorTest, SystemInfoHandlerTest, TestExclusionRuleCollectionAccess, SampleTest, TestQueryTypes, TestFuzzyAnalyzedSuggestions, BigEndianAscendingWordSerializerTest, TestSQLHandlerNonCloud, TestTolerantUpdateProcessorRandomCloud, SolrCLIZkUtilsTest, TestSimpleQParserPlugin, FileUtilsTest, TestLRUCache, CachingDirectoryFactoryTest, DateMathParserTest, URLClassifyProcessorTest, TestSubQueryTransformerCrossCore, DistributedQueryComponentCustomSortTest, SolrIndexSplitterTest, TestJsonFacetRefinement, SortByFunctionTest, TestNoOpRegenerator, DocValuesNotIndexedTest, DeleteNodeTest, TestSolrQueryResponse, LeaderInitiatedRecoveryOnCommitTest]
   [junit4] Completed [113/662 (1!)] on J1 in 39.03s, 1 test, 1 error <<< FAILURES!

[...truncated 64209 lines...]



Mime
View raw message