lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 236 - Failure
Date Sat, 24 Dec 2016 16:00:19 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/236/

3 tests failed.
FAILED:  org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([BE3B25C06784B95A:C5AFBF7FE54498ED]:0)
	at java.util.HashMap.newNode(HashMap.java:1742)
	at java.util.HashMap.putVal(HashMap.java:641)
	at java.util.HashMap.put(HashMap.java:611)
	at java.util.HashSet.add(HashSet.java:219)
	at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:104)
	at org.apache.lucene.store.MockDirectoryWrapper.listAll(MockDirectoryWrapper.java:1037)
	at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)
	at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)
	at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2239)
	at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2181)
	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1133)
	at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1175)
	at org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:301)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testCollectionsAPI

Error Message:
Expected to see collection awhollynewcollection_0 null Last available state: DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/6)={   "replicationFactor":"4",   "shards":{     "shard1":{       "range":"80000000-ffffffff",       "state":"active",       "replicas":{}},     "shard2":{       "range":"0-7fffffff",       "state":"active",       "replicas":{}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"3",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected to see collection awhollynewcollection_0
null
Last available state: DocCollection(awhollynewcollection_0//collections/awhollynewcollection_0/state.json/6)={
  "replicationFactor":"4",
  "shards":{
    "shard1":{
      "range":"80000000-ffffffff",
      "state":"active",
      "replicas":{}},
    "shard2":{
      "range":"0-7fffffff",
      "state":"active",
      "replicas":{}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"3",
  "autoAddReplicas":"false"}
	at __randomizedtesting.SeedInfo.seed([15CEDE5379D39D0D:5DBBAAE77FE0B298]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:265)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:496)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.testSpecificConfigsets

Error Message:
KeeperErrorCode = NoNode for /collections/withconfigset2

Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/withconfigset2
	at __randomizedtesting.SeedInfo.seed([15CEDE5379D39D0D:38B091098EF72701]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:356)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:353)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:353)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:425)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 993 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexingSequenceNumbers
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexingSequenceNumbers -Dtests.method=testStressConcurrentCommit -Dtests.seed=BE3B25C06784B95A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/test-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=America/Nipigon -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1079s J0 | TestIndexingSequenceNumbers.testStressConcurrentCommit <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BE3B25C06784B95A:C5AFBF7FE54498ED]:0)
   [junit4]    > 	at java.util.HashMap.newNode(HashMap.java:1742)
   [junit4]    > 	at java.util.HashMap.putVal(HashMap.java:641)
   [junit4]    > 	at java.util.HashMap.put(HashMap.java:611)
   [junit4]    > 	at java.util.HashSet.add(HashSet.java:219)
   [junit4]    > 	at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:104)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.listAll(MockDirectoryWrapper.java:1037)
   [junit4]    > 	at org.apache.lucene.store.FilterDirectory.listAll(FilterDirectory.java:57)
   [junit4]    > 	at org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:428)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2239)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2181)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1133)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1175)
   [junit4]    > 	at org.apache.lucene.index.TestIndexingSequenceNumbers.testStressConcurrentCommit(TestIndexingSequenceNumbers.java:301)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/J0/temp/lucene.index.TestIndexingSequenceNumbers_BE3B25C06784B95A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {id=FST50}, docValues:{thread=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=2027, maxMBSortInHeap=6.545992085514463, sim=RandomSimilarity(queryNorm=false,coord=no): {id=DFR I(n)3(800.0)}, locale=hr, timezone=America/Nipigon
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=1,free=74458504,total=495976448
   [junit4]   2> NOTE: All tests run in this JVM: [TestSmallFloat, LimitedFiniteStringsIteratorTest, TestDirectMonotonic, TestWordlistLoader, TestCustomNorms, TestIndexWriterMergePolicy, TestFieldType, TestBooleanCoord, TestFixedBitDocIdSet, TestTwoPhaseCommitTool, TestTimSorter, TestIndexWriterOnVMError, TestDocValuesIndexing, TestNRTReaderWithThreads, TestOmitPositions, TestBytesRef, TestPhraseQuery, TestCharFilter, TestMultiPhraseQuery, TestIndexingSequenceNumbers]
   [junit4] Completed [220/442 (1!)] on J0 in 1304.15s, 8 tests, 1 error <<< FAILURES!

[...truncated 732 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/lucene/build/core/test/temp/junit4-J0-20161224_112004_338.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/heapdumps/java_pid12132.hprof ...
   [junit4] Heap dump file created [555311893 bytes in 2.703 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 11184 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/init-core-data-001
   [junit4]   2> 4226026 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 4226027 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001
   [junit4]   2> 4226027 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4226031 INFO  (Thread-51930) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4226031 INFO  (Thread-51930) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4226132 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:35605
   [junit4]   2> 4226141 INFO  (jetty-launcher-4817-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 4226143 INFO  (jetty-launcher-4817-thread-3) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 4226144 INFO  (jetty-launcher-4817-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 4226145 INFO  (jetty-launcher-4817-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@b29b436{/solr,null,AVAILABLE}
   [junit4]   2> 4226147 INFO  (jetty-launcher-4817-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@f435327{SSL,[ssl, http/1.1]}{127.0.0.1:50303}
   [junit4]   2> 4226147 INFO  (jetty-launcher-4817-thread-2) [    ] o.e.j.s.Server Started @4229925ms
   [junit4]   2> 4226147 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=50303}
   [junit4]   2> 4226147 ERROR (jetty-launcher-4817-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4226148 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.4.0
   [junit4]   2> 4226148 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4226148 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4226148 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-24T15:41:38.582Z
   [junit4]   2> 4226148 INFO  (jetty-launcher-4817-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5b2fd475{/solr,null,AVAILABLE}
   [junit4]   2> 4226150 INFO  (jetty-launcher-4817-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7ec131ee{SSL,[ssl, http/1.1]}{127.0.0.1:34894}
   [junit4]   2> 4226150 INFO  (jetty-launcher-4817-thread-1) [    ] o.e.j.s.Server Started @4229928ms
   [junit4]   2> 4226150 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34894}
   [junit4]   2> 4226151 ERROR (jetty-launcher-4817-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4226151 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.4.0
   [junit4]   2> 4226151 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4226151 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4226151 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-24T15:41:38.585Z
   [junit4]   2> 4226151 INFO  (jetty-launcher-4817-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4e51bd22{/solr,null,AVAILABLE}
   [junit4]   2> 4226153 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4226153 INFO  (jetty-launcher-4817-thread-3) [    ] o.e.j.s.AbstractConnector Started ServerConnector@6243c3fe{SSL,[ssl, http/1.1]}{127.0.0.1:59127}
   [junit4]   2> 4226153 INFO  (jetty-launcher-4817-thread-3) [    ] o.e.j.s.Server Started @4229931ms
   [junit4]   2> 4226153 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59127}
   [junit4]   2> 4226153 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4226158 ERROR (jetty-launcher-4817-thread-3) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4226158 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.4.0
   [junit4]   2> 4226158 INFO  (jetty-launcher-4817-thread-4) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 4226158 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4226158 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4226159 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-24T15:41:38.593Z
   [junit4]   2> 4226161 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4226162 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4226165 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4226166 INFO  (jetty-launcher-4817-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35605/solr
   [junit4]   2> 4226167 INFO  (jetty-launcher-4817-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35605/solr
   [junit4]   2> 4226168 INFO  (jetty-launcher-4817-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@8bce84d{/solr,null,AVAILABLE}
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-4) [    ] o.e.j.s.AbstractConnector Started ServerConnector@77f282ad{SSL,[ssl, http/1.1]}{127.0.0.1:41982}
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-4) [    ] o.e.j.s.Server Started @4229949ms
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41982}
   [junit4]   2> 4226171 ERROR (jetty-launcher-4817-thread-4) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35605/solr
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr™ version 6.4.0
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 4226171 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 4226172 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2016-12-24T15:41:38.606Z
   [junit4]   2> 4226172 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x159317e9d220004, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4226174 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x159317e9d220006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4226175 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x159317e9d220005, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 4226177 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 4226182 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4226183 INFO  (jetty-launcher-4817-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35605/solr
   [junit4]   2> 4226212 INFO  (jetty-launcher-4817-thread-1) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34894_solr
   [junit4]   2> 4226216 INFO  (jetty-launcher-4817-thread-3) [n:127.0.0.1:59127_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59127_solr
   [junit4]   2> 4226216 INFO  (jetty-launcher-4817-thread-1) [n:127.0.0.1:34894_solr    ] o.a.s.c.Overseer Overseer (id=97163286837067785-127.0.0.1:34894_solr-n_0000000000) starting
   [junit4]   2> 4226216 INFO  (jetty-launcher-4817-thread-4) [n:127.0.0.1:41982_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41982_solr
   [junit4]   2> 4226217 INFO  (jetty-launcher-4817-thread-2) [n:127.0.0.1:50303_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50303_solr
   [junit4]   2> 4226218 INFO  (zkCallback-4833-thread-1-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4226226 INFO  (zkCallback-4831-thread-1-processing-n:127.0.0.1:50303_solr) [n:127.0.0.1:50303_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 4226226 INFO  (zkCallback-4834-thread-1-processing-n:127.0.0.1:59127_solr) [n:127.0.0.1:59127_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4226226 INFO  (zkCallback-4838-thread-1-processing-n:127.0.0.1:41982_solr) [n:127.0.0.1:41982_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 4226229 INFO  (zkCallback-4831-thread-2-processing-n:127.0.0.1:50303_solr) [n:127.0.0.1:50303_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4226229 INFO  (zkCallback-4833-thread-1-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 4226253 INFO  (jetty-launcher-4817-thread-1) [n:127.0.0.1:34894_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34894_solr
   [junit4]   2> 4226254 INFO  (zkCallback-4834-thread-1-processing-n:127.0.0.1:59127_solr) [n:127.0.0.1:59127_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4226254 INFO  (zkCallback-4838-thread-1-processing-n:127.0.0.1:41982_solr) [n:127.0.0.1:41982_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4226255 INFO  (zkCallback-4831-thread-2-processing-n:127.0.0.1:50303_solr) [n:127.0.0.1:50303_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4226258 INFO  (jetty-launcher-4817-thread-3) [n:127.0.0.1:59127_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001/node3/.
   [junit4]   2> 4226259 INFO  (zkCallback-4833-thread-1-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 4226275 INFO  (jetty-launcher-4817-thread-4) [n:127.0.0.1:41982_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001/node4/.
   [junit4]   2> 4226308 INFO  (jetty-launcher-4817-thread-1) [n:127.0.0.1:34894_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001/node1/.
   [junit4]   2> 4226375 INFO  (jetty-launcher-4817-thread-2) [n:127.0.0.1:50303_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001/node2/.
   [junit4]   2> 4226380 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4226381 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35605/solr ready
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 4226424 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 4226439 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4226441 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4226455 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_49824_hdfs____.fx8dj5/webapp
   [junit4]   2> 4226871 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49824
   [junit4]   2> 4226959 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4226960 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4226974 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_56716_datanode____.k5ff3y/webapp
   [junit4]   2> 4227379 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56716
   [junit4]   2> 4227432 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4227436 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4227461 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.2-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36786_datanode____imdb89/webapp
   [junit4]   2> 4227518 INFO  (IPC Server handler 5 on 44124) [    ] BlockStateChange BLOCK* processReport: from storage DS-9e1107aa-ad8e-4ffb-a42c-9cfc809dfd9a node DatanodeRegistration(127.0.0.1:49908, datanodeUuid=bb82dd9d-9a6a-4dc5-84d9-94b29565624d, infoPort=41892, infoSecurePort=0, ipcPort=36893, storageInfo=lv=-56;cid=testClusterID;nsid=1907367615;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 4227518 INFO  (IPC Server handler 5 on 44124) [    ] BlockStateChange BLOCK* processReport: from storage DS-2f78cb15-eb71-4065-b8c9-cededc399b6b node DatanodeRegistration(127.0.0.1:49908, datanodeUuid=bb82dd9d-9a6a-4dc5-84d9-94b29565624d, infoPort=41892, infoSecurePort=0, ipcPort=36893, storageInfo=lv=-56;cid=testClusterID;nsid=1907367615;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 4227893 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36786
   [junit4]   2> 4227990 INFO  (IPC Server handler 4 on 44124) [    ] BlockStateChange BLOCK* processReport: from storage DS-f0f5d9f0-51e8-41bc-a047-e62712ff4357 node DatanodeRegistration(127.0.0.1:39529, datanodeUuid=4e4ea2bc-a7dd-4b28-b295-ee86a86647cf, infoPort=41487, infoSecurePort=0, ipcPort=33128, storageInfo=lv=-56;cid=testClusterID;nsid=1907367615;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
   [junit4]   2> 4227990 INFO  (IPC Server handler 4 on 44124) [    ] BlockStateChange BLOCK* processReport: from storage DS-6bf68e04-1e61-4cdb-a9f7-374233efbb14 node DatanodeRegistration(127.0.0.1:39529, datanodeUuid=4e4ea2bc-a7dd-4b28-b295-ee86a86647cf, infoPort=41487, infoSecurePort=0, ipcPort=33128, storageInfo=lv=-56;cid=testClusterID;nsid=1907367615;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
   [junit4]   2> 4228096 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[15CEDE5379D39D0D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testCreationAndDeletion
   [junit4]   2> 4228097 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[15CEDE5379D39D0D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4228157 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=created_and_deleted&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4228160 INFO  (OverseerThreadFactory-10819-thread-1-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.CreateCollectionCmd Create collection created_and_deleted
   [junit4]   2> 4228325 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2
   [junit4]   2> 4228431 INFO  (zkCallback-4833-thread-2-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4229336 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 4229358 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.s.IndexSchema [created_and_deleted_shard1_replica1] Schema name=minimal
   [junit4]   2> 4229361 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 4229362 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'created_and_deleted_shard1_replica1' using configuration from collection created_and_deleted
   [junit4]   2> 4229369 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:44124/data
   [junit4]   2> 4229369 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 4229369 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.SolrCore [[created_and_deleted_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001/node1/created_and_deleted_shard1_replica1], dataDir=[hdfs://localhost:44124/data/created_and_deleted/core_node1/data/]
   [junit4]   2> 4229369 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/created_and_deleted/core_node1/data/snapshot_metadata
   [junit4]   2> 4229378 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4229378 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 4229378 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4229388 WARN  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 4234560 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4234561 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/created_and_deleted/core_node1/data
   [junit4]   2> 4234577 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/created_and_deleted/core_node1/data/index
   [junit4]   2> 4234584 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4234584 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 4234584 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4234600 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4234627 INFO  (IPC Server handler 0 on 44124) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49908 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6bf68e04-1e61-4cdb-a9f7-374233efbb14:NORMAL:127.0.0.1:39529|RBW], ReplicaUC[[DISK]DS-2f78cb15-eb71-4065-b8c9-cededc399b6b:NORMAL:127.0.0.1:49908|RBW]]} size 0
   [junit4]   2> 4234628 INFO  (IPC Server handler 2 on 44124) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39529 is added to blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6bf68e04-1e61-4cdb-a9f7-374233efbb14:NORMAL:127.0.0.1:39529|RBW], ReplicaUC[[DISK]DS-2f78cb15-eb71-4065-b8c9-cededc399b6b:NORMAL:127.0.0.1:49908|RBW]]} size 0
   [junit4]   2> 4234643 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 4234643 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4234643 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 4234663 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4234663 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4234673 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1349bec5[created_and_deleted_shard1_replica1] main]
   [junit4]   2> 4234674 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 4234674 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 4234674 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4234675 INFO  (searcherExecutor-10828-thread-1-processing-n:127.0.0.1:34894_solr x:created_and_deleted_shard1_replica1 s:shard1 c:created_and_deleted) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.SolrCore [created_and_deleted_shard1_replica1] Registered new searcher Searcher@1349bec5[created_and_deleted_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4234675 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1554612598455926784
   [junit4]   2> 4234681 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4234681 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4234681 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:34894/solr/created_and_deleted_shard1_replica1/
   [junit4]   2> 4234681 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4234681 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:34894/solr/created_and_deleted_shard1_replica1/ has no replicas
   [junit4]   2> 4234683 INFO  (zkCallback-4833-thread-3-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4234686 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:34894/solr/created_and_deleted_shard1_replica1/ shard1
   [junit4]   2> 4234790 INFO  (zkCallback-4833-thread-3-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4234838 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4234840 INFO  (qtp170159908-70883) [n:127.0.0.1:34894_solr c:created_and_deleted s:shard1  x:created_and_deleted_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=created_and_deleted_shard1_replica1&action=CREATE&numShards=1&collection=created_and_deleted&shard=shard1&wt=javabin&version=2} status=0 QTime=6514
   [junit4]   2> 4234842 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 4234942 INFO  (zkCallback-4833-thread-2-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/created_and_deleted/state.json] for collection [created_and_deleted] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4235843 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=created_and_deleted&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=7685
   [junit4]   2> 4235845 INFO  (qtp1444828252-70897) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4235846 INFO  (qtp1444828252-70897) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 4235847 INFO  (qtp1444828252-70889) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=created_and_deleted&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4235849 INFO  (OverseerThreadFactory-10819-thread-2-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 4235850 INFO  (OverseerCollectionConfigSetProcessor-97163286837067785-127.0.0.1:34894_solr-n_0000000000) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4235852 INFO  (qtp170159908-70884) [n:127.0.0.1:34894_solr    ] o.a.s.c.SolrCore [created_and_deleted_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@a841a33
   [junit4]   2> 4235856 INFO  (qtp170159908-70884) [n:127.0.0.1:34894_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/created_and_deleted/core_node1/data/snapshot_metadata
   [junit4]   2> 4235857 INFO  (qtp170159908-70884) [n:127.0.0.1:34894_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/created_and_deleted/core_node1/data/index
   [junit4]   2> 4235857 INFO  (qtp170159908-70884) [n:127.0.0.1:34894_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/created_and_deleted/core_node1/data
   [junit4]   2> 4235866 INFO  (IPC Server handler 5 on 44124) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741825_1001 127.0.0.1:39529 127.0.0.1:49908 
   [junit4]   2> 4235868 INFO  (qtp170159908-70884) [n:127.0.0.1:34894_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=created_and_deleted_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=17
   [junit4]   2> 4235944 INFO  (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@5546dd0) [    ] BlockStateChange BLOCK* BlockManager: ask 127.0.0.1:39529 to delete [blk_1073741825_1001]
   [junit4]   2> 4236574 INFO  (qtp1444828252-70889) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=created_and_deleted&action=DELETE&wt=javabin&version=2} status=0 QTime=726
   [junit4]   2> 4236576 INFO  (qtp1444828252-70893) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4236576 INFO  (qtp1444828252-70893) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 4236576 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testCreationAndDeletion-seed#[15CEDE5379D39D0D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testCreationAndDeletion
   [junit4]   2> 4236656 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[15CEDE5379D39D0D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testTooManyReplicas
   [junit4]   2> 4236657 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[15CEDE5379D39D0D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4236661 INFO  (qtp1444828252-70893) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4236663 INFO  (OverseerThreadFactory-10819-thread-3-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection
   [junit4]   2> 4236663 WARN  (OverseerThreadFactory-10819-thread-3-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.CreateCollectionCmd Specified replicationFactor of 10 on collection collection is higher than or equal to the number of Solr instances currently live or live and part of your createNodeSet(4). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 4236663 INFO  (OverseerCollectionConfigSetProcessor-97163286837067785-127.0.0.1:34894_solr-n_0000000000) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4236664 ERROR (OverseerThreadFactory-10819-thread-3-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerCollectionMessageHandler Collection: collection operation: create failed:org.apache.solr.common.SolrException: Cannot create collection collection. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 4. This allows a maximum of 4 to be created. Value of numShards is 2 and value of replicationFactor is 10. This requires 20 shards to be created (higher than the allowed number)
   [junit4]   2> 	at org.apache.solr.cloud.CreateCollectionCmd.call(CreateCollectionCmd.java:155)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:224)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:463)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 4236666 INFO  (qtp1444828252-70893) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=10&collection.configName=conf&name=collection&action=CREATE&numShards=2&wt=javabin&version=2} status=400 QTime=5
   [junit4]   2> 4236667 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testTooManyReplicas-seed#[15CEDE5379D39D0D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testTooManyReplicas
   [junit4]   2> 4236743 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk-seed#[15CEDE5379D39D0D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting deleteCollectionOnlyInZk
   [junit4]   2> 4236745 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk-seed#[15CEDE5379D39D0D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4236750 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=onlyinzk&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4236753 INFO  (OverseerCollectionConfigSetProcessor-97163286837067785-127.0.0.1:34894_solr-n_0000000000) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4236766 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=onlyinzk&action=DELETE&wt=javabin&version=2} status=0 QTime=15
   [junit4]   2> 4236768 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4236768 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 4236769 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=onlyinzk&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4236775 INFO  (OverseerThreadFactory-10819-thread-5-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.CreateCollectionCmd Create collection onlyinzk
   [junit4]   2> 4236775 INFO  (OverseerCollectionConfigSetProcessor-97163286837067785-127.0.0.1:34894_solr-n_0000000000) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4236986 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=onlyinzk_shard2_replica1&action=CREATE&numShards=2&collection=onlyinzk&shard=shard2&wt=javabin&version=2
   [junit4]   2> 4236986 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf&newCollection=true&name=onlyinzk_shard1_replica1&action=CREATE&numShards=2&collection=onlyinzk&shard=shard1&wt=javabin&version=2
   [junit4]   2> 4237093 INFO  (zkCallback-4833-thread-1-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4237093 INFO  (zkCallback-4834-thread-2-processing-n:127.0.0.1:59127_solr) [n:127.0.0.1:59127_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4237996 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 4237996 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.4.0
   [junit4]   2> 4238005 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.s.IndexSchema [onlyinzk_shard2_replica1] Schema name=minimal
   [junit4]   2> 4238007 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.s.IndexSchema [onlyinzk_shard1_replica1] Schema name=minimal
   [junit4]   2> 4238008 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 4238009 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 4238009 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'onlyinzk_shard2_replica1' using configuration from collection onlyinzk
   [junit4]   2> 4238010 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'onlyinzk_shard1_replica1' using configuration from collection onlyinzk
   [junit4]   2> 4238010 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:44124/data
   [junit4]   2> 4238010 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 4238010 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.SolrCore [[onlyinzk_shard2_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001/node1/onlyinzk_shard2_replica1], dataDir=[hdfs://localhost:44124/data/onlyinzk/core_node1/data/]
   [junit4]   2> 4238010 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:44124/data
   [junit4]   2> 4238010 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/onlyinzk/core_node1/data/snapshot_metadata
   [junit4]   2> 4238010 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 4238010 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.SolrCore [[onlyinzk_shard1_replica1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001/tempDir-001/node3/onlyinzk_shard1_replica1], dataDir=[hdfs://localhost:44124/data/onlyinzk/core_node2/data/]
   [junit4]   2> 4238010 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/onlyinzk/core_node2/data/snapshot_metadata
   [junit4]   2> 4238019 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4238019 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4238019 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 4238019 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 4238019 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4238019 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4238030 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4238030 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4238031 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/onlyinzk/core_node1/data
   [junit4]   2> 4238031 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/onlyinzk/core_node2/data
   [junit4]   2> 4238050 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/onlyinzk/core_node1/data/index
   [junit4]   2> 4238051 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44124/data/onlyinzk/core_node2/data/index
   [junit4]   2> 4238057 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4238057 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 4238058 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4238058 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 4238058 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 4238058 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 4238072 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4238072 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 4238094 INFO  (IPC Server handler 4 on 44124) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49908 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6bf68e04-1e61-4cdb-a9f7-374233efbb14:NORMAL:127.0.0.1:39529|RBW], ReplicaUC[[DISK]DS-9e1107aa-ad8e-4ffb-a42c-9cfc809dfd9a:NORMAL:127.0.0.1:49908|RBW]]} size 0
   [junit4]   2> 4238095 INFO  (IPC Server handler 2 on 44124) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39529 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-9e1107aa-ad8e-4ffb-a42c-9cfc809dfd9a:NORMAL:127.0.0.1:49908|RBW], ReplicaUC[[DISK]DS-f0f5d9f0-51e8-41bc-a047-e62712ff4357:NORMAL:127.0.0.1:39529|FINALIZED]]} size 0
   [junit4]   2> 4238096 INFO  (IPC Server handler 8 on 44124) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49908 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6bf68e04-1e61-4cdb-a9f7-374233efbb14:NORMAL:127.0.0.1:39529|RBW], ReplicaUC[[DISK]DS-2f78cb15-eb71-4065-b8c9-cededc399b6b:NORMAL:127.0.0.1:49908|RBW]]} size 0
   [junit4]   2> 4238097 INFO  (IPC Server handler 7 on 44124) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39529 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6bf68e04-1e61-4cdb-a9f7-374233efbb14:NORMAL:127.0.0.1:39529|RBW], ReplicaUC[[DISK]DS-2f78cb15-eb71-4065-b8c9-cededc399b6b:NORMAL:127.0.0.1:49908|RBW]]} size 0
   [junit4]   2> 4238121 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 4238121 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4238121 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 4238127 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 4238127 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4238127 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=3
   [junit4]   2> 4238133 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4238133 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4238137 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4238137 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4238147 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6fd75977[onlyinzk_shard2_replica1] main]
   [junit4]   2> 4238148 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 4238149 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 4238149 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4238149 INFO  (searcherExecutor-10833-thread-1-processing-n:127.0.0.1:34894_solr x:onlyinzk_shard2_replica1 s:shard2 c:onlyinzk) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.SolrCore [onlyinzk_shard2_replica1] Registered new searcher Searcher@6fd75977[onlyinzk_shard2_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4238149 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1554612602098679808
   [junit4]   2> 4238151 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e391290[onlyinzk_shard1_replica1] main]
   [junit4]   2> 4238152 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
   [junit4]   2> 4238153 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
   [junit4]   2> 4238153 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4238153 INFO  (searcherExecutor-10834-thread-1-processing-n:127.0.0.1:59127_solr x:onlyinzk_shard1_replica1 s:shard1 c:onlyinzk) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.SolrCore [onlyinzk_shard1_replica1] Registered new searcher Searcher@3e391290[onlyinzk_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4238153 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1554612602102874112
   [junit4]   2> 4238159 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4238159 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4238159 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:34894/solr/onlyinzk_shard2_replica1/
   [junit4]   2> 4238159 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4238160 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:34894/solr/onlyinzk_shard2_replica1/ has no replicas
   [junit4]   2> 4238162 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4238162 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4238162 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:59127/solr/onlyinzk_shard1_replica1/
   [junit4]   2> 4238162 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4238162 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.SyncStrategy https://127.0.0.1:59127/solr/onlyinzk_shard1_replica1/ has no replicas
   [junit4]   2> 4238164 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:34894/solr/onlyinzk_shard2_replica1/ shard2
   [junit4]   2> 4238166 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:59127/solr/onlyinzk_shard1_replica1/ shard1
   [junit4]   2> 4238268 INFO  (zkCallback-4833-thread-1-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4238268 INFO  (zkCallback-4834-thread-2-processing-n:127.0.0.1:59127_solr) [n:127.0.0.1:59127_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4238315 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4238317 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 4238317 INFO  (qtp170159908-70877) [n:127.0.0.1:34894_solr c:onlyinzk s:shard2  x:onlyinzk_shard2_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=onlyinzk_shard2_replica1&action=CREATE&numShards=2&collection=onlyinzk&shard=shard2&wt=javabin&version=2} status=0 QTime=1331
   [junit4]   2> 4238318 INFO  (qtp1444828252-70895) [n:127.0.0.1:59127_solr c:onlyinzk s:shard1  x:onlyinzk_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf&newCollection=true&name=onlyinzk_shard1_replica1&action=CREATE&numShards=2&collection=onlyinzk&shard=shard1&wt=javabin&version=2} status=0 QTime=1332
   [junit4]   2> 4238321 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 4238423 INFO  (zkCallback-4833-thread-3-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4238423 INFO  (zkCallback-4834-thread-2-processing-n:127.0.0.1:59127_solr) [n:127.0.0.1:59127_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/onlyinzk/state.json] for collection [onlyinzk] has occurred - updating... (live nodes size: [4])
   [junit4]   2> 4238778 INFO  (OverseerCollectionConfigSetProcessor-97163286837067785-127.0.0.1:34894_solr-n_0000000000) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000008 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 4238944 INFO  (org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@5546dd0) [    ] BlockStateChange BLOCK* BlockManager: ask 127.0.0.1:49908 to delete [blk_1073741825_1001]
   [junit4]   2> 4239321 INFO  (qtp1444828252-70896) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=onlyinzk&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2551
   [junit4]   2> 4239323 INFO  (qtp1444828252-70889) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4239323 INFO  (qtp1444828252-70889) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 4239323 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.deleteCollectionOnlyInZk-seed#[15CEDE5379D39D0D]) [    ] o.a.s.SolrTestCaseJ4 ###Ending deleteCollectionOnlyInZk
   [junit4]   2> 4239401 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[15CEDE5379D39D0D]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMissingRequiredParameters
   [junit4]   2> 4239402 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.testMissingRequiredParameters-seed#[15CEDE5379D39D0D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 4239405 INFO  (qtp1444828252-70889) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=onlyinzk&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4239407 INFO  (OverseerThreadFactory-10819-thread-5-processing-n:127.0.0.1:34894_solr) [n:127.0.0.1:34894_solr    ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
   [junit4]   2> 4239409 INFO  (qtp170159908-70880) [n:127.0.0.1:34894_solr    ] o.a.s.c.SolrCore [onlyinzk_shard2_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@56b59170
   [junit4]   2> 4239409 INFO  (qtp1444828252-70897) [n:127.0.0.1:59127_solr    ] o.a.s.c.SolrCore [onlyinzk_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1533e029
   [junit4]   2> 4239415 INFO  (qtp1444828252-70897) [n:127.0.0.1:59127_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/onlyinzk/core_node2/data/snapshot_metadata
   [junit4]   2> 4239415 INFO  (qtp1444828252-70897) [n:127.0.0.1:59127_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/onlyinzk/core_node2/data
   [junit4]   2> 4239418 INFO  (qtp1444828252-70897) [n:127.0.0.1:59127_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/onlyinzk/core_node2/data/index
   [junit4]   2> 4239418 INFO  (qtp170159908-70880) [n:127.0.0.1:34894_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/onlyinzk/core_node1/data/snapshot_metadata
   [junit4]   2> 4239423 INFO  (qtp170159908-70880) [n:127.0.0.1:34894_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/onlyinzk/core_node1/data/index
   [junit4]   2> 4239423 INFO  (qtp170159908-70880) [n:127.0.0.1:34894_solr    ] o.a.s.s.h.HdfsDirectory Closing hdfs directory hdfs://localhost:44124/data/onlyinzk/core_node1/data
   [junit4]   2> 4239446 INFO  (IPC Server handler 3 on 44124) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741826_1002 127.0.0.1:39529 127.0.0.1:49908 
   [junit4]   2> 4239450 INFO  (qtp1444828252-70897) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=onlyinzk_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=42
   [junit4]   2> 4239452 INFO  (IPC Server handler 0 on 44124) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741827_1003 127.0.0.1:39529 127.0.0.1:49908 
   [junit4]   2> 4239454 INFO  (qtp170159908-70880) [n:127.0.0.1:34894_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=onlyinzk_shard2_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=45
   [junit4]   2> 4240158 INFO  (qtp1444828252-70889) [n:127.0.0.1:59127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=onlyinzk&action=DELETE&wt=javabin&version=2} status=0 QTime=753
   [junit4]   2> 4240160 INFO  (qtp1444828252-70894) [n:127.0.0.1:59127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 4240161 ERROR (qtp1444828252-70894) [n:127.0.0.1:59127_solr    ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Missing required parameter: name
   [junit4]   2> 	at org.apache.solr.common.params.RequiredSolrParams.getParams(RequiredSolrParams.java:90)
   [junit4]   2> 	at org.apache.solr.common.params.SolrParams.getAll(SolrParams.java:461)
   [junit4]   2> 	at org.apache.solr.common.params.SolrParams.getAll(SolrParams.java:478)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$0(CollectionsHandler.java:378)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:872)
   [junit4]   2> 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:209)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:152)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:664)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:318)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:110)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1699)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:534)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:202)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4

[...truncated too long message...]

s.server.namenode.NameNode.stop(NameNode.java:872)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:52)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:203)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3370)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5729)
   [junit4]   2> 	... 54 more
   [junit4]   2> 4387901 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35605 35605
   [junit4]   2> 4387901 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[15CEDE5379D39D0D]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_15CEDE5379D39D0D-001
   [junit4]   2> Dec 24, 2016 3:44:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 35 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=nl-NL, timezone=America/Montreal
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=4,threads=2,free=315741392,total=514326528
   [junit4]   2> NOTE: All tests run in this JVM: [TestReversedWildcardFilterFactory, TestCloudDeleteByQuery, TestShardHandlerFactory, SolrPluginUtilsTest, TestCollectionAPI, BlobRepositoryCloudTest, PrimUtilsTest, TestSolrJ, DistributedQueueTest, AtomicUpdatesTest, TestSha256AuthenticationProvider, HardAutoCommitTest, TestSubQueryTransformerDistrib, TestSortByMinMaxFunction, TestLeaderInitiatedRecoveryThread, ShowFileRequestHandlerTest, TestGroupingSearch, TestExtendedDismaxParser, TestDFISimilarityFactory, TestLFUCache, TestReloadDeadlock, ResponseLogComponentTest, SaslZkACLProviderTest, DirectUpdateHandlerTest, TestAuthorizationFramework, CircularListTest, TestDistribIDF, SolrCoreCheckLockOnStartupTest, ConnectionReuseTest, HdfsBasicDistributedZk2Test, RecoveryZkTest, ManagedSchemaRoundRobinCloudTest, OpenExchangeRatesOrgProviderTest, OutOfBoxZkACLAndCredentialsProvidersTest, DirectUpdateHandlerOptimizeTest, QueryParsingTest, TestBulkSchemaConcurrent, DocValuesMultiTest, TestJsonFacets, ForceLeaderTest, HdfsChaosMonkeySafeLeaderTest, LukeRequestHandlerTest, DOMUtilTest, SuggesterTSTTest, MinimalSchemaTest, CoreAdminHandlerTest, HLLSerializationTest, TestFieldCacheSort, UtilsToolTest, HdfsDirectoryTest, TestSolrConfigHandlerCloud, TestSolrXml, TestIntervalFaceting, TestSolrQueryParser, DeleteInactiveReplicaTest, TestGraphTermsQParserPlugin, TestReload, TestShortCircuitedRequests, TestRestoreCore, NoCacheHeaderTest, DebugComponentTest, TestMaxScoreQueryParser, XmlUpdateRequestHandlerTest, OverseerTest, TestDFRSimilarityFactory, ConjunctionSolrSpellCheckerTest, TestUseDocValuesAsStored, TestRebalanceLeaders, TestOverriddenPrefixQueryForCustomFieldType, TestIndexingPerformance, DistributedQueryComponentOptimizationTest, CurrencyFieldXmlFileTest, SolrTestCaseJ4Test, TestNumericTerms64, DistributedFacetPivotLargeTest, TestZkChroot, SharedFSAutoReplicaFailoverUtilsTest, TestRestManager, TestRTimerTree, TestSolrDeletionPolicy1, DateRangeFieldTest, TestCloudSchemaless, TestFieldCache, CoreSorterTest, FullSolrCloudDistribCmdsTest, CdcrBootstrapTest, CollectionsAPISolrJTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestPostingsSolrHighlighter, TestDynamicLoading, TermVectorComponentDistributedTest, TestLRUStatsCache, AnalysisErrorHandlingTest, TestCustomSort, HdfsUnloadDistributedZkTest, TestFieldTypeCollectionResource, SolrCloudExampleTest, TestFieldCacheWithThreads, PeerSyncWithIndexFingerprintCachingTest, DistribCursorPagingTest, TestConfigSets, TestGraphMLResponseWriter, PrimitiveFieldTypeTest, LeaderFailoverAfterPartitionTest, BadCopyFieldTest, TestOnReconnectListenerSupport, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, TestReplicationHandler, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, SpellCheckComponentTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFiltering, TestBadConfig, DistributedQueryElevationComponentTest, SpellCheckCollatorTest, SuggesterTest, TestTrie, SpatialFilterTest, SuggesterWFSTTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, SolrInfoMBeanTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, DirectSolrSpellCheckerTest, TestQueryTypes, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, FieldAnalysisRequestHandlerTest, LoggingHandlerTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestBinaryResponseWriter, SOLR749Test, TestQuerySenderListener, AlternateDirectoryTest, TestQuerySenderNoQuery, SolrIndexConfigTest, SampleTest, TestBinaryField, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestLuceneMatchVersion, TestPhraseSuggestions, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestJmxMonitoredMap, PluginInfoTest, TestSystemIdResolver, TestSuggestSpellingConverter, DistributedMLTComponentTest, TestHighlightDedupGrouping, TestTolerantSearch, ActionThrottleTest, AssignTest, CdcrReplicationHandlerTest, ConfigSetsAPITest, ConnectionManagerTest, CreateCollectionCleanupTest, DeleteNodeTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, LeaderInitiatedRecoveryOnCommitTest, LeaderInitiatedRecoveryOnShardRestartTest, OverseerModifyCollectionTest, OverseerStatusTest, OverseerTaskQueueTest, RemoteQueryErrorTest, ReplicationFactorTest, RestartWhileUpdatingTest, ShardSplitTest, TestTolerantUpdateProcessorRandomCloud, TriLevelCompositeIdRoutingTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [533/665 (1!)] on J2 in 171.02s, 20 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 12046 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/build.xml:783: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/checkout/build.xml:727: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid12132.hprof

Total time: 279 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




Mime
View raw message