lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 1061 - Still Failing
Date Wed, 30 Dec 2015 17:35:48 GMT
I'll raise the time on this for nightly runs.

- Mark

On Tue, Dec 29, 2015 at 6:47 PM Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/1061/
>
> 3 tests failed.
> FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest.test
>
> Error Message:
> The Monkey ran for over 45 seconds and no jetties were stopped - this is
> worth investigating!
>
> Stack Trace:
> java.lang.AssertionError: The Monkey ran for over 45 seconds and no
> jetties were stopped - this is worth investigating!
>         at
> __randomizedtesting.SeedInfo.seed([EA3B02E14E63D573:626F3D3BE09FB88B]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:560)
>         at
> org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:146)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
>         at
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:965)
>         at
> org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:940)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED:  junit.framework.TestSuite.org
> .apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
>
> Error Message:
> 2 threads leaked from SUITE scope at
> org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:     1)
> Thread[id=15765, name=StoppableIndexingThread, state=TIMED_WAITING,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at
> java.lang.Thread.sleep(Native Method)         at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:136)
>   2) Thread[id=15764, name=StoppableIndexingThread, state=TIMED_WAITING,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at
> java.lang.Thread.sleep(Native Method)         at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:136)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from
> SUITE scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest:
>    1) Thread[id=15765, name=StoppableIndexingThread, state=TIMED_WAITING,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]
>         at java.lang.Thread.sleep(Native Method)
>         at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:136)
>    2) Thread[id=15764, name=StoppableIndexingThread, state=TIMED_WAITING,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]
>         at java.lang.Thread.sleep(Native Method)
>         at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:136)
>         at __randomizedtesting.SeedInfo.seed([EA3B02E14E63D573]:0)
>
>
> FAILED:  junit.framework.TestSuite.org
> .apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
>
> Error Message:
> There are still zombie threads that couldn't be terminated:    1)
> Thread[id=15764, name=StoppableIndexingThread, state=RUNNABLE,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at
> java.lang.Integer.getChars(Integer.java:372)         at
> java.lang.Integer.toString(Integer.java:332)         at
> java.lang.Integer.toString(Integer.java:739)         at
> org.apache.solr.client.solrj.util.ClientUtils.writeVal(ClientUtils.java:164)
>        at
> org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:132)
>        at
> org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:413)
>        at
> org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:329)
>        at
> org.apache.solr.client.solrj.request.RequestWriter.getContentStream(RequestWriter.java:67)
>        at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:95)
>        at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:105)
>        at
> org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:356)
>        at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
>        at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
>        at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
>    at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:167)
>    at
> org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:177)
>        at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:116)
>   2) Thread[id=15765, name=StoppableIndexingThread, state=RUNNABLE,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]         at
> java.lang.Integer.getChars(Integer.java:372)         at
> java.lang.Integer.toString(Integer.java:332)         at
> java.lang.Integer.toString(Integer.java:739)         at
> org.apache.solr.client.solrj.util.ClientUtils.writeVal(ClientUtils.java:164)
>        at
> org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:132)
>        at
> org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:413)
>        at
> org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:329)
>        at
> org.apache.solr.client.solrj.request.RequestWriter.getContentStream(RequestWriter.java:67)
>        at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:95)
>        at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:105)
>        at
> org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:356)
>        at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
>        at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
>        at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
>    at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:167)
>    at
> org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:177)
>        at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:116)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie
> threads that couldn't be terminated:
>    1) Thread[id=15764, name=StoppableIndexingThread, state=RUNNABLE,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]
>         at java.lang.Integer.getChars(Integer.java:372)
>         at java.lang.Integer.toString(Integer.java:332)
>         at java.lang.Integer.toString(Integer.java:739)
>         at
> org.apache.solr.client.solrj.util.ClientUtils.writeVal(ClientUtils.java:164)
>         at
> org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:132)
>         at
> org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:413)
>         at
> org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:329)
>         at
> org.apache.solr.client.solrj.request.RequestWriter.getContentStream(RequestWriter.java:67)
>         at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:95)
>         at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:105)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:356)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
>         at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
>         at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:167)
>         at
> org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:177)
>         at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:116)
>    2) Thread[id=15765, name=StoppableIndexingThread, state=RUNNABLE,
> group=TGRP-HdfsChaosMonkeySafeLeaderTest]
>         at java.lang.Integer.getChars(Integer.java:372)
>         at java.lang.Integer.toString(Integer.java:332)
>         at java.lang.Integer.toString(Integer.java:739)
>         at
> org.apache.solr.client.solrj.util.ClientUtils.writeVal(ClientUtils.java:164)
>         at
> org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:132)
>         at
> org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:413)
>         at
> org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:329)
>         at
> org.apache.solr.client.solrj.request.RequestWriter.getContentStream(RequestWriter.java:67)
>         at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:95)
>         at
> org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:105)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.createMethod(HttpSolrClient.java:356)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:239)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
>         at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
>         at
> org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:167)
>         at
> org.apache.solr.cloud.StoppableIndexingThread.indexDocs(StoppableIndexingThread.java:177)
>         at
> org.apache.solr.cloud.StoppableIndexingThread.run(StoppableIndexingThread.java:116)
>         at __randomizedtesting.SeedInfo.seed([EA3B02E14E63D573]:0)
>
>
>
>
> Build Log:
> [...truncated 11542 lines...]
>    [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
>    [junit4]   2> Creating dataDir:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/init-core-data-001
>    [junit4]   2> 3024380 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
>    [junit4]   2> 3024380 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.a.s.BaseDistributedSearchTestCase Setting hostContext system property:
> /rz/r
>    [junit4]   1> Formatting using clusterid: testClusterID
>    [junit4]   2> 3024425 WARN
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.a.h.m.i.MetricsConfig Cannot locate configuration: tried
> hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
>    [junit4]   2> 3024434 WARN
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
>    [junit4]   2> 3024437 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log jetty-6.1.26
>    [junit4]   2> 3024450 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log Extract
> jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs
> to ./temp/Jetty_localhost_40848_hdfs____.j54i3a/webapp
>    [junit4]   2> 3024574 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log NO JSP Support for /, did not find
> org.apache.jasper.servlet.JspServlet
>    [junit4]   2> 3024791 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost
> :40848
>    [junit4]   2> 3024870 WARN
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
>    [junit4]   2> 3024871 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log jetty-6.1.26
>    [junit4]   2> 3024886 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log Extract
> jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
> to ./temp/Jetty_localhost_58802_datanode____fny2am/webapp
>    [junit4]   2> 3025001 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log NO JSP Support for /, did not find
> org.apache.jasper.servlet.JspServlet
>    [junit4]   2> 3025322 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost
> :58802
>    [junit4]   2> 3025358 WARN
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
>    [junit4]   2> 3025360 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log jetty-6.1.26
>    [junit4]   2> 3025373 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log Extract
> jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode
> to ./temp/Jetty_localhost_51889_datanode____.q33tks/webapp
>    [junit4]   2> 3025406 INFO  (IPC Server handler 0 on 50235) [    ]
> BlockStateChange BLOCK* processReport: from storage
> DS-1fad80c6-90eb-46db-9c7d-86281455e07f node
> DatanodeRegistration(127.0.0.1,
> datanodeUuid=10f01f83-63a4-4333-9074-5abef760e258, infoPort=58802,
> ipcPort=54385, storageInfo=lv=-56;cid=testClusterID;nsid=1611814338;c=0),
> blocks: 0, hasStaleStorages: true, processing time: 0 msecs
>    [junit4]   2> 3025406 INFO  (IPC Server handler 0 on 50235) [    ]
> BlockStateChange BLOCK* processReport: from storage
> DS-40d84459-703f-40dc-8e56-8cc0330ada3e node
> DatanodeRegistration(127.0.0.1,
> datanodeUuid=10f01f83-63a4-4333-9074-5abef760e258, infoPort=58802,
> ipcPort=54385, storageInfo=lv=-56;cid=testClusterID;nsid=1611814338;c=0),
> blocks: 0, hasStaleStorages: false, processing time: 0 msecs
>    [junit4]   2> 3025499 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log NO JSP Support for /, did not find
> org.apache.jasper.servlet.JspServlet
>    [junit4]   2> 3025706 INFO
> (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[EA3B02E14E63D573]-worker) [    ]
> o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost
> :51889
>    [junit4]   2> 3025803 INFO  (IPC Server handler 8 on 50235) [    ]
> BlockStateChange BLOCK* processReport: from storage
> DS-a361307c-b633-4137-b8a8-4e378c07432e node
> DatanodeRegistration(127.0.0.1,
> datanodeUuid=e4622669-05f4-4517-8b93-a00030a5b52a, infoPort=51889,
> ipcPort=57183, storageInfo=lv=-56;cid=testClusterID;nsid=1611814338;c=0),
> blocks: 0, hasStaleStorages: true, processing time: 1 msecs
>    [junit4]   2> 3025803 INFO  (IPC Server handler 8 on 50235) [    ]
> BlockStateChange BLOCK* processReport: from storage
> DS-11a4f7cc-a74a-49ea-be8b-584a47ab345d node
> DatanodeRegistration(127.0.0.1,
> datanodeUuid=e4622669-05f4-4517-8b93-a00030a5b52a, infoPort=51889,
> ipcPort=57183, storageInfo=lv=-56;cid=testClusterID;nsid=1611814338;c=0),
> blocks: 0, hasStaleStorages: false, processing time: 0 msecs
>    [junit4]   2> 3025887 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>    [junit4]   2> 3025888 INFO  (Thread-6976) [    ] o.a.s.c.ZkTestServer
> client port:0.0.0.0/0.0.0.0:0
>    [junit4]   2> 3025888 INFO  (Thread-6976) [    ] o.a.s.c.ZkTestServer
> Starting server
>    [junit4]   2> 3025988 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.ZkTestServer start zk server on port:59312
>    [junit4]   2> 3025988 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>    [junit4]   2> 3025989 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>    [junit4]   2> 3025992 INFO  (zkCallback-2011-thread-1) [    ]
> o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@36a71322
> name:ZooKeeperConnection Watcher:127.0.0.1:59312 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 3025992 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>    [junit4]   2> 3025992 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>    [junit4]   2> 3025992 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /solr
>    [junit4]   2> 3025995 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>    [junit4]   2> 3025995 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>    [junit4]   2> 3025996 INFO  (zkCallback-2012-thread-1) [    ]
> o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@629fafa0
> name:ZooKeeperConnection Watcher:127.0.0.1:59312/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 3025996 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>    [junit4]   2> 3025997 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>    [junit4]   2> 3025997 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /collections/collection1
>    [junit4]   2> 3025998 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
>    [junit4]   2> 3026000 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
>    [junit4]   2> 3026001 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
>    [junit4]   2> 3026002 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
>    [junit4]   2> 3026002 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
>    [junit4]   2> 3026005 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml
> to /configs/conf1/schema.xml
>    [junit4]   2> 3026005 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
>    [junit4]   2> 3026006 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 3026007 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 3026008 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
>    [junit4]   2> 3026008 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
>    [junit4]   2> 3026010 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
>    [junit4]   2> 3026010 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
>    [junit4]   2> 3026011 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
>    [junit4]   2> 3026012 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
>    [junit4]   2> 3026013 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
>    [junit4]   2> 3026013 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
>    [junit4]   2> 3026015 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 3026015 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 3026017 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 3026017 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 3026018 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
>    [junit4]   2> 3026018 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
>    [junit4]   2> 3026020 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.AbstractZkTestCase put
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
>    [junit4]   2> 3026020 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
>    [junit4]   2> 3026315 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.SolrTestCaseJ4 Writing core.properties file to
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/cores/collection1
>    [junit4]   2> 3026317 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.e.j.s.Server jetty-9.2.13.v20150730
>    [junit4]   2> 3026319 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4c048f9a
> {/rz/r,null,AVAILABLE}
>    [junit4]   2> 3026319 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.e.j.s.ServerConnector Started ServerConnector@20dfb26a{HTTP/1.1}{
> 127.0.0.1:49192}
>    [junit4]   2> 3026319 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.e.j.s.Server Started @3028954ms
>    [junit4]   2> 3026320 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.s.e.JettySolrRunner Jetty properties:
> {solr.data.dir=hdfs://localhost:50235/hdfs__localhost_50235__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-5.x_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001_tempDir-002_control_data,
> hostPort=49192,
> coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/cores,
> hostContext=/rz/r}
>    [junit4]   2> 3026320 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init():
> sun.misc.Launcher$AppClassLoader@3827de24
>    [junit4]   2> 3026320 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory:
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001'
>    [junit4]   2> 3026320 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 3026320 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find
> system property or JNDI)
>    [junit4]   2> 3026320 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>    [junit4]   2> 3026324 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>    [junit4]   2> 3026325 INFO  (zkCallback-2013-thread-1) [    ]
> o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@4b42d03
> name:ZooKeeperConnection Watcher:127.0.0.1:59312/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 3026325 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>    [junit4]   2> 3026325 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>    [junit4]   2> 3026326 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in
> ZooKeeper)
>    [junit4]   2> 3026326 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.SolrXmlConfig Loading container configuration from
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/solr.xml
>    [junit4]   2> 3026340 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.CorePropertiesLocator Config-defined core root directory:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/cores
>    [junit4]   2> 3026340 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.CoreContainer New CoreContainer 1615142391
>    [junit4]   2> 3026340 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.CoreContainer Loading cores into CoreContainer
> [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001]
>    [junit4]   2> 3026340 WARN
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.CoreContainer Couldn't add files from
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/lib
> to classpath:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/lib
>    [junit4]   2> 3026340 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout :
> 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost :
> 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize :
> 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy :
> false,useRetries : false,
>    [junit4]   2> 3026342 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with
> params: socketTimeout=340000&connTimeout=45000&retry=true
>    [junit4]   2> 3026342 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 3026342 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.l.LogWatcher Registering Log Listener [Log4j
> (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 3026342 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59312/solr
>    [junit4]   2> 3026342 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.ZkController zkHost includes chroot
>    [junit4]   2> 3026342 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>    [junit4]   2> 3026343 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>    [junit4]   2> 3026344 INFO  (zkCallback-2015-thread-1) [    ]
> o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@6364e675
> name:ZooKeeperConnection Watcher:127.0.0.1:59312 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 3026344 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>    [junit4]   2> 3026345 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>    [junit4]   2> 3026346 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.ConnectionManager Waiting for
> client to connect to ZooKeeper
>    [junit4]   2> 3026347 INFO
> (zkCallback-2016-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@707b168b
> name:ZooKeeperConnection Watcher:127.0.0.1:59312/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 3026347 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.ConnectionManager Client is
> connected to ZooKeeper
>    [junit4]   2> 3026348 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer/queue
>    [junit4]   2> 3026350 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer/collection-queue-work
>    [junit4]   2> 3026353 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer/collection-map-running
>    [junit4]   2> 3026355 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer/collection-map-completed
>    [junit4]   2> 3026356 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer/collection-map-failure
>    [junit4]   2> 3026358 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /live_nodes
>    [junit4]   2> 3026359 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /aliases.json
>    [junit4]   2> 3026361 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /clusterstate.json
>    [junit4]   2> 3026362 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /security.json
>    [junit4]   2> 3026362 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.ZkController Register node as live
> in ZooKeeper:/live_nodes/127.0.0.1:49192_rz%2Fr
>    [junit4]   2> 3026363 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /live_nodes/127.0.0.1:49192_rz%2Fr
>    [junit4]   2> 3026364 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer_elect
>    [junit4]   2> 3026365 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer_elect/election
>    [junit4]   2> 3026366 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.Overseer Overseer (id=null) closing
>    [junit4]   2> 3026367 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.LeaderElector Joined leadership
> election with path:
> /overseer_elect/election/95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000
>    [junit4]   2> 3026367 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.OverseerElectionContext I am going
> to be the leader 127.0.0.1:49192_rz%2Fr
>    [junit4]   2> 3026367 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer_elect/leader
>    [junit4]   2> 3026369 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.Overseer Overseer
> (id=95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000) starting
>    [junit4]   2> 3026370 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.SolrZkClient makePath:
> /overseer/queue-work
>    [junit4]   2> 3026375 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.OverseerAutoReplicaFailoverThread
> Starting OverseerAutoReplicaFailoverThread
> autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
>    [junit4]   2> 3026376 INFO
> (OverseerCollectionConfigSetProcessor-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.OverseerTaskProcessor Process
> current queue of overseer operations
>    [junit4]   2> 3026376 INFO
> (OverseerStateUpdate-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.Overseer Starting to work on the
> main queue
>    [junit4]   2> 3026376 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.ZkStateReader Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 3026405 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.CoreContainer Security conf
> doesn't exist. Skipping setup for authorization module.
>    [junit4]   2> 3026405 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.CoreContainer No authentication
> plugin used.
>    [junit4]   2> 3026406 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.CorePropertiesLocator Looking for
> core definitions underneath
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/cores
>    [junit4]   2> 3026407 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.CoreDescriptor Created
> CoreDescriptor: {dataDir=data/, loadOnStartup=true,
> collection=control_collection, shard=, transient=false,
> config=solrconfig.xml, schema=schema.xml, name=collection1,
> configSetProperties=configsetprops.json, coreNodeName=}
>    [junit4]   2> 3026407 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.CorePropertiesLocator Found core
> collection1 in
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/cores/collection1
>    [junit4]   2> 3026407 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.CorePropertiesLocator Found 1 core
> definitions
>    [junit4]   2> 3026408 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.s.SolrDispatchFilter
> user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
>    [junit4]   2> 3026408 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.ZkController publishing state=down
>    [junit4]   2> 3026408 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573])
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.s.SolrDispatchFilter
> SolrDispatchFilter.init() done
>    [junit4]   2> 3026408 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.ZkController numShards not found on descriptor - reading it from
> system property
>    [junit4]   2> 3026410 INFO
> (OverseerStateUpdate-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.Overseer processMessage:
> queueSize: 1, message = {
>    [junit4]   2>   "shard":null,
>    [junit4]   2>   "node_name":"127.0.0.1:49192_rz%2Fr",
>    [junit4]   2>   "core":"collection1",
>    [junit4]   2>   "roles":null,
>    [junit4]   2>   "operation":"state",
>    [junit4]   2>   "base_url":"http://127.0.0.1:49192/rz/r",
>    [junit4]   2>   "numShards":"1",
>    [junit4]   2>   "collection":"control_collection",
>    [junit4]   2>   "state":"down"} current state version: 0
>    [junit4]   2> 3026410 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.ZkController look for our core node name
>    [junit4]   2> 3026410 INFO
> (OverseerStateUpdate-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.o.ReplicaMutator Update state
> numShards=1 message={
>    [junit4]   2>   "shard":null,
>    [junit4]   2>   "node_name":"127.0.0.1:49192_rz%2Fr",
>    [junit4]   2>   "core":"collection1",
>    [junit4]   2>   "roles":null,
>    [junit4]   2>   "operation":"state",
>    [junit4]   2>   "base_url":"http://127.0.0.1:49192/rz/r",
>    [junit4]   2>   "numShards":"1",
>    [junit4]   2>   "collection":"control_collection",
>    [junit4]   2>   "state":"down"}
>    [junit4]   2> 3026411 INFO
> (OverseerStateUpdate-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.o.ClusterStateMutator building a
> new cName: control_collection
>    [junit4]   2> 3026411 INFO
> (OverseerStateUpdate-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.o.ReplicaMutator Assigning new
> node to shard shard=shard1
>    [junit4]   2> 3026412 INFO
> (zkCallback-2016-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 3027411 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.ZkController waiting to find shard id in clusterstate for
> collection1
>    [junit4]   2> 3027411 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.ZkController Check for collection zkNode:control_collection
>    [junit4]   2> 3027411 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.ZkController Collection zkNode exists
>    [junit4]   2> 3027411 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.c.ZkStateReader Load collection config from:
> [/collections/control_collection]
>    [junit4]   2> 3027412 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.c.ZkStateReader path=[/collections/control_collection]
> [configName]=[conf1] specified config exists in ZooKeeper
>    [junit4]   2> 3027412 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory:
> '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/cores/collection1'
>    [junit4]   2> 3027412 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>    [junit4]   2> 3027412 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find
> system property or JNDI)
>    [junit4]   2> 3027414 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.Config loaded config solrconfig.xml with version 0
>    [junit4]   2> 3027419 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.SolrConfig current version of requestparams : -1
>    [junit4]   2> 3027431 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.SolrConfig Using Lucene MatchVersion: 5.5.0
>    [junit4]   2> 3027457 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
>    [junit4]   2> 3027464 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.s.IndexSchema [collection1] Schema name=test
>    [junit4]   2> 3027658 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.s.IndexSchema default search field in schema is text
>    [junit4]   2> 3027660 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.s.IndexSchema unique key field: id
>    [junit4]   2> 3027661 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file
> currency.xml
>    [junit4]   2> 3027664 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file
> currency.xml
>    [junit4]   2> 3027678 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming
> default properties: Can't find resource 'configsetprops.json' in classpath
> or '/configs/conf1',
> cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1
>    [junit4]   2> 3027679 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection   x:collection1]
> o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration
> from collection control_collection
>    [junit4]   2> 3027679 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
>    [junit4]   2> 3027679 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.HdfsDirectoryFactory
> solr.hdfs.home=hdfs://localhost:50235/solr_hdfs_home
>    [junit4]   2> 3027679 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication
> disabled
>    [junit4]   2> 3027679 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at
> [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_EA3B02E14E63D573-001/control-001/cores/collection1],
> dataDir=[hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data/]
>    [junit4]   2> 3027679 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding
> Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11c0600a
>    [junit4]   2> 3027680 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for
> path
> hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data
>    [junit4]   2> 3027698 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for
> hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data
>    [junit4]   2> 3027699 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrCore New index directory detected: old=null
> new=hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data/index/
>    [junit4]   2> 3027709 WARN
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory
> 'hdfs:/localhost:50235/solr_hdfs_home/control_collection/core_node1/data/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 3027710 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for
> path
> hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data/index
>    [junit4]   2> 3027718 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache
> [1] with direct memory allocation set to [true]
>    [junit4]   2> 3027718 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory
> usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608]
> bytes
>    [junit4]   2> 3027718 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS
> BlockCache
>    [junit4]   2> 3027728 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
>    [junit4]   2> 3027728 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for
> hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data/index
>    [junit4]   2> 3027729 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5070751795047811]
>    [junit4]   2> 3027748 INFO  (IPC Server handler 9 on 50235) [    ]
> BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38880
> is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-40d84459-703f-40dc-8e56-8cc0330ada3e:NORMAL:127.0.0.1:46787|RBW],
> ReplicaUnderConstruction[[DISK]DS-a361307c-b633-4137-b8a8-4e378c07432e:NORMAL:127.0.0.1:38880|RBW]]}
> size 0
>    [junit4]   2> 3027750 INFO  (IPC Server handler 2 on 50235) [    ]
> BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46787
> is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION,
> primaryNodeIndex=-1,
> replicas=[ReplicaUnderConstruction[[DISK]DS-40d84459-703f-40dc-8e56-8cc0330ada3e:NORMAL:127.0.0.1:46787|RBW],
> ReplicaUnderConstruction[[DISK]DS-a361307c-b633-4137-b8a8-4e378c07432e:NORMAL:127.0.0.1:38880|RBW]]}
> size 0
>    [junit4]   2> 3027753 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit:
> commits: num=1
>    [junit4]   2>        commit{dir=NRTCachingDirectory(BlockDirectory(
> HdfsDirectory@hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data/index
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@70755032);
> maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
>    [junit4]   2> 3027753 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>    [junit4]   2> 3027755 INFO
> (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore
> Looking for old index directories to cleanup for core collection1 in
> hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data/
>    [junit4]   2> 3027758 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating
> updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 3027758 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 3027758 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>    [junit4]   2> 3027758 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 3027758 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
>    [junit4]   2> 3027758 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating
> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 3027759 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 3027759 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
>    [junit4]   2> 3027759 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as
> default, creating implicit default
>    [junit4]   2> 3027759 WARN
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler
> {type = requestHandler,name = /dump,class = DumpRequestHandler,args =
> {defaults={a=A,b=B}}}
>    [junit4]   2> 3027762 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>    [junit4]   2> 3027763 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>    [junit4]   2> 3027764 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>    [junit4]   2> 3027765 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
>    [junit4]   2> 3027768 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.RequestHandlers Registered paths:
> /update/json/docs,/admin/properties,/get,/update/json,/schema,/dump,/update,/admin/plugins,/admin/mbeans,/admin/system,/admin/logging,/admin/ping,/replication,/admin/file,/update/csv,standard,/admin/segments,/admin/luke,/admin/threads,/config
>    [junit4]   2> 3027770 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrCore Using default statsCache cache:
> org.apache.solr.search.stats.LocalStatsCache
>    [junit4]   2> 3027770 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation:
> org.apache.solr.update.HdfsUpdateLog
>    [junit4]   2> 3027770 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: dataDir=
> defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
> tlogDfsReplication=2
>    [junit4]   2> 3027782 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for
> 15000ms;
>    [junit4]   2> 3027782 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
>    [junit4]   2> 3027784 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class
> org.apache.lucene.index.MockRandomMergePolicy:
> org.apache.lucene.index.MockRandomMergePolicy@7835e1fd
>    [junit4]   2> 3027793 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit:
> commits: num=1
>    [junit4]   2>        commit{dir=NRTCachingDirectory(BlockDirectory(
> HdfsDirectory@hdfs://localhost:50235/solr_hdfs_home/control_collection/core_node1/data/index
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@70755032);
> maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
>    [junit4]   2> 3027793 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>    [junit4]   2> 3027793 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@197e4ac8[collection1]
> main
>    [junit4]   2> 3027793 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.c.ZkStateReader Load collection config from:
> [/collections/control_collection]
>    [junit4]   2> 3027794 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.c.ZkStateReader
> path=[/collections/control_collection] [configName]=[conf1] specified
> config exists in ZooKeeper
>    [junit4]   2> 3027794 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based
> storage for the RestManager with znodeBase: /configs/conf1
>    [junit4]   2> 3027794 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO
> with znodeBase: /configs/conf1
>    [junit4]   2> 3027794 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs:
> {}
>    [junit4]   2> 3027795 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json
> using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 3027795 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode
> /configs/conf1/_rest_managed.json
>    [junit4]   2> 3027795 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 3027795 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.r.RestManager Initializing 0 registered
> ManagedResources
>    [junit4]   2> 3027795 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for
> 10000
>    [junit4]   2> 3027797 INFO
> (searcherExecutor-6607-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore
> [collection1] Registered new searcher Searcher@197e4ac8[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 3027798 WARN
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.s.h.HdfsLocalityReporter Could not retrieve locality
> information for hdfs://localhost:36278/solr3 due to exception:
> java.net.ConnectException: Call From lucene1-us-west/10.41.0.5 to
> localhost:36278 failed on connection exception: java.net.ConnectException:
> Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
>    [junit4]   2> 3027798 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to
> seed version buckets
>    [junit4]   2> 3027798 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_
> for 256 version buckets from index
>    [junit4]   2> 3027798 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot
> seed version bucket highest value from index
>    [junit4]   2> 3027798 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.UpdateLog Could not find max version in index or
> recent updates, using new clock 1521936908906659840
>    [junit4]   2> 3027798 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.u.UpdateLog Took 0.0ms to seed version buckets with
> highest version 1521936908906659840
>    [junit4]   2> 3027798 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
>    [junit4]   2> 3027799 INFO
> (coreLoadExecutor-6606-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr c:control_collection s:shard1 r:core_node1
> x:collection1] o.a.s.c.CoreContainer registering core: collection1
>    [junit4]   2> 3027799 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.ZkController Register replica - core:collection1 address:
> http://127.0.0.1:49192/rz/r collection:control_collection shard:shard1
>    [junit4]   2> 3027800 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.c.SolrZkClient makePath:
> /collections/control_collection/leader_elect/shard1/election
>    [junit4]   2> 3027803 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.LeaderElector Joined leadership election with path:
> /collections/control_collection/leader_elect/shard1/election/95121056681558020-core_node1-n_0000000000
>    [junit4]   2> 3027804 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.ShardLeaderElectionContext Running the leader process for
> shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
>    [junit4]   2> 3027805 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
>    [junit4]   2> 3027805 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
>    [junit4]   2> 3027805 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.SyncStrategy Sync replicas to
> http://127.0.0.1:49192/rz/r/collection1/
>    [junit4]   2> 3027805 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
>    [junit4]   2> 3027805 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.SyncStrategy http://127.0.0.1:49192/rz/r/collection1/ has no
> replicas
>    [junit4]   2> 3027805 INFO
> (OverseerStateUpdate-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.Overseer processMessage:
> queueSize: 1, message = {
>    [junit4]   2>   "operation":"leader",
>    [junit4]   2>   "shard":"shard1",
>    [junit4]   2>   "collection":"control_collection"} current state
> version: 1
>    [junit4]   2> 3027806 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.c.SolrZkClient makePath:
> /collections/control_collection/leaders/shard1
>    [junit4]   2> 3027808 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
>    [junit4]   2> 3027810 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:control_collection s:shard1 r:core_node1 x:collection1]
> o.a.s.c.ShardLeaderElectionContext I am the new leader:
> http://127.0.0.1:49192/rz/r/collection1/ shard1
>    [junit4]   2> 3027810 INFO
> (OverseerStateUpdate-95121056681558020-127.0.0.1:49192_rz%2Fr-n_0000000000)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.Overseer processMessage:
> queueSize: 1, message = {
>    [junit4]   2>   "operation":"leader",
>    [junit4]   2>   "shard":"shard1",
>    [junit4]   2>   "collection":"control_collection",
>    [junit4]   2>   "base_url":"http://127.0.0.1:49192/rz/r",
>    [junit4]   2>   "core":"collection1",
>    [junit4]   2>   "state":"active"} current state version: 1
>    [junit4]   2> 3027910 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>    [junit4]   2> 3027911 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>    [junit4]   2> 3027912 INFO
> (zkCallback-2016-thread-1-processing-n:127.0.0.1:49192_rz%2Fr)
> [n:127.0.0.1:49192_rz%2Fr    ] o.a.s.c.c.ZkStateReader A cluster state
> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
>    [junit4]   2> 3027913 INFO  (zkCallback-2018-thread-1) [    ]
> o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@6a282588
> name:ZooKeeperConnection Watcher:127.0.0.1:59312/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 3027913 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>    [junit4]   2> 3027913 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>    [junit4]   2> 3027913 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper...
>    [junit4]   2> 3027915 INFO
> (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[EA3B02E14E63D573]) [    ]
> o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection
> loss:false
>    [junit4]   2> 3027961 INFO
> (coreZkRegister-6600-thread-1-processing-s:shard1 n:127.0.0.1:49192_rz%2Fr
> x:collection1 r:core_node1 c:control_collection) [n:127.0.0.1:49192_rz%2Fr
> c:

-- 
- Mark
about.me/markrmiller

Mime
View raw message