lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 707 - Still Failing
Date Thu, 18 Dec 2014 21:23:10 GMT
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/707/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=1502 sum(shards)=1503 cloudClient=1503

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=1502 sum(shards)=1503 cloudClient=1503
	at __randomizedtesting.SeedInfo.seed([AFE8A0D51025B96:8B180415265D3BAA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1308)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:232)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
expected:<...E8A0D51025B96-001/te[mpDir-003/solrj_test_core_props_shard1_replica1]> but was:<...E8A0D51025B96-001/te[stPropertyParamsForCreate-001/instanceDir-p]>

Stack Trace:
org.junit.ComparisonFailure: expected:<...E8A0D51025B96-001/te[mpDir-003/solrj_test_core_props_shard1_replica1]> but was:<...E8A0D51025B96-001/te[stPropertyParamsForCreate-001/instanceDir-p]>
	at __randomizedtesting.SeedInfo.seed([AFE8A0D51025B96:8B180415265D3BAA]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.checkInstanceDirs(CollectionsAPIDistributedZkTest.java:1136)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:883)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:204)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10245 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/init-core-data-001
   [junit4]   2> 1546775 T10980 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1546775 T10980 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1546782 T10980 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1546783 T10980 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1546783 T10981 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1546883 T10980 oasc.ZkTestServer.run start zk server on port:33371
   [junit4]   2> 1546884 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1546885 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1546889 T10988 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74b60bd4 name:ZooKeeperConnection Watcher:127.0.0.1:33371 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1546889 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1546890 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1546890 T10980 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1546892 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1546893 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1546895 T10991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@164dbf9e name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1546895 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1546895 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1546896 T10980 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1546898 T10980 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1546899 T10980 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1546900 T10980 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1546902 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1546903 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1546905 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1546906 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1547008 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1547008 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1547010 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1547011 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1547013 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1547013 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1547015 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1547016 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1547017 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1547018 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1547020 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1547020 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1547022 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1547023 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1547024 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1547025 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1547027 T10980 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1547027 T10980 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1547029 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1547030 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1547032 T10994 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2611491e name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1547032 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1547032 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 1548740 T10980 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1548745 T10980 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1548747 T10980 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:40340
   [junit4]   2> 1548749 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1548749 T10980 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1548749 T10980 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002
   [junit4]   2> 1548750 T10980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/'
   [junit4]   2> 1548794 T10980 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solr.xml
   [junit4]   2> 1548870 T10980 oasc.CoreContainer.<init> New CoreContainer 2018228790
   [junit4]   2> 1548871 T10980 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/]
   [junit4]   2> 1548871 T10980 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1548872 T10980 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1548872 T10980 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1548872 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1548872 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1548873 T10980 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1548873 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1548873 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1548873 T10980 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1548873 T10980 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1548875 T10980 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1548877 T10980 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1548877 T10980 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1548877 T10980 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1548878 T10980 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33371/solr
   [junit4]   2> 1548878 T10980 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1548878 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1548879 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1548880 T11006 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f04f95 name:ZooKeeperConnection Watcher:127.0.0.1:33371 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1548881 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1548881 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1548882 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1548883 T11009 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@191a2ac0 name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1548883 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1548884 T10980 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1548885 T10980 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1548886 T10980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1548888 T10980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1548889 T10980 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1548890 T10980 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1548891 T10980 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40340_
   [junit4]   2> 1548892 T10980 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40340_
   [junit4]   2> 1548893 T10980 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1548894 T10980 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1548894 T10980 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1548895 T10980 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40340_
   [junit4]   2> 1548896 T10980 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1548897 T10980 oasc.Overseer.start Overseer (id=92991275417600004-127.0.0.1:40340_-n_0000000000) starting
   [junit4]   2> 1548898 T10980 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1548901 T10980 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1548902 T11011 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1548902 T10980 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1548904 T10980 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1548904 T10980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1548906 T11010 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1548909 T10980 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1548909 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1548912 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1548912 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1548913 T11016 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d5ce18 name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1548914 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1548914 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1548914 T10980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1548916 T10980 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1548916 T10980 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1548916 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1548917 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1548918 T11019 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c506a9d name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1548918 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1548918 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1548919 T11009 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1548921 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 0
   [junit4]   2> 1548921 T11010 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 1548922 T11010 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1548923 T11009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1548923 T11016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1549841 T10980 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1549841 T10980 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1549845 T10980 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1549846 T10980 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:30763
   [junit4]   2> 1549848 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1549849 T10980 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1549849 T10980 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-003
   [junit4]   2> 1549849 T10980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-003/'
   [junit4]   2> 1549909 T10980 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-003/solr.xml
   [junit4]   2> 1549960 T10980 oasc.CoreContainer.<init> New CoreContainer 2136406851
   [junit4]   2> 1549961 T10980 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-003/]
   [junit4]   2> 1549962 T10980 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1549962 T10980 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1549962 T10980 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1549962 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1549963 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1549963 T10980 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1549963 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1549963 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1549964 T10980 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1549964 T10980 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1549966 T10980 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1549967 T10980 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1549967 T10980 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1549968 T10980 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1549968 T10980 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33371/solr
   [junit4]   2> 1549968 T10980 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1549968 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1549969 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1549971 T11031 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ff8f754 name:ZooKeeperConnection Watcher:127.0.0.1:33371 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1549972 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1549972 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1549974 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1549975 T11034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@553cbe37 name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1549975 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1549980 T10980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1550984 T10980 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30763_
   [junit4]   2> 1550985 T10980 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30763_
   [junit4]   2> 1550988 T10980 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1550992 T10980 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1550993 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1552181 T10980 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1552182 T10980 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1552185 T10980 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1552187 T10980 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:18175
   [junit4]   2> 1552189 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1552189 T10980 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1552189 T10980 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-004
   [junit4]   2> 1552189 T10980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-004/'
   [junit4]   2> 1552219 T10980 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-004/solr.xml
   [junit4]   2> 1552282 T10980 oasc.CoreContainer.<init> New CoreContainer 1950129491
   [junit4]   2> 1552282 T10980 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-004/]
   [junit4]   2> 1552283 T10980 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1552283 T10980 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1552284 T10980 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1552284 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1552284 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1552284 T10980 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1552284 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1552285 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1552285 T10980 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1552285 T10980 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1552287 T10980 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1552288 T10980 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1552289 T10980 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1552289 T10980 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1552289 T10980 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33371/solr
   [junit4]   2> 1552290 T10980 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1552290 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1552290 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1552292 T11047 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11969360 name:ZooKeeperConnection Watcher:127.0.0.1:33371 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1552293 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1552294 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1552295 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1552297 T11050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d2c2506 name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1552297 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1552301 T10980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1553306 T10980 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:18175_
   [junit4]   2> 1553307 T10980 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:18175_
   [junit4]   2> 1553310 T10980 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1553315 T10980 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1553315 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1554528 T10980 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1554530 T10980 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1554533 T10980 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1554535 T10980 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:58828
   [junit4]   2> 1554536 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1554537 T10980 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1554537 T10980 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-005
   [junit4]   2> 1554537 T10980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-005/'
   [junit4]   2> 1554567 T10980 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-005/solr.xml
   [junit4]   2> 1554619 T10980 oasc.CoreContainer.<init> New CoreContainer 85765973
   [junit4]   2> 1554620 T10980 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-005/]
   [junit4]   2> 1554621 T10980 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1554621 T10980 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1554621 T10980 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1554621 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1554622 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1554622 T10980 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1554622 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1554622 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1554622 T10980 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1554623 T10980 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1554624 T10980 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1554626 T10980 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1554626 T10980 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1554626 T10980 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1554626 T10980 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33371/solr
   [junit4]   2> 1554627 T10980 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1554627 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1554628 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1554629 T11063 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12e9f354 name:ZooKeeperConnection Watcher:127.0.0.1:33371 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1554630 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1554631 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1554632 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1554633 T11066 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b3877e9 name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1554634 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1554638 T10980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1555642 T10980 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58828_
   [junit4]   2> 1555643 T10980 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58828_
   [junit4]   2> 1555646 T10980 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1555651 T10980 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1555652 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1556755 T10980 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1556756 T10980 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1556760 T10980 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1556761 T10980 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:38904
   [junit4]   2> 1556763 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1556763 T10980 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1556763 T10980 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-006
   [junit4]   2> 1556764 T10980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-006/'
   [junit4]   2> 1556793 T10980 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-006/solr.xml
   [junit4]   2> 1556845 T10980 oasc.CoreContainer.<init> New CoreContainer 843877745
   [junit4]   2> 1556845 T10980 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-006/]
   [junit4]   2> 1556846 T10980 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1556846 T10980 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1556846 T10980 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1556846 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1556847 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1556847 T10980 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1556847 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1556847 T10980 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1556848 T10980 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1556848 T10980 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1556850 T10980 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1556851 T10980 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1556851 T10980 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1556851 T10980 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1556852 T10980 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33371/solr
   [junit4]   2> 1556852 T10980 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1556852 T10980 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1556853 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1556854 T11079 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33fe0cf name:ZooKeeperConnection Watcher:127.0.0.1:33371 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1556856 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1556856 T10980 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1556857 T10980 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1556859 T11082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50da400f name:ZooKeeperConnection Watcher:127.0.0.1:33371/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1556859 T10980 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1556863 T10980 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1557867 T10980 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38904_
   [junit4]   2> 1557868 T10980 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38904_
   [junit4]   2> 1557872 T10980 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1557876 T10980 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1557877 T10980 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1558007 T11022 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&name=solrj_collection&action=CREATE&numShards=2&wt=javabin&collection.configName=conf1&version=2&router.field=myOwnField
   [junit4]   2> 1558008 T11009 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 1558009 T11011 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1558010 T11085 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1558010 T11085 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1558011 T11085 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1558013 T11009 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1558014 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"} current state version: 1
   [junit4]   2> 1558014 T11010 oasco.ClusterStateMutator.createCollection building a new cName: solrj_collection
   [junit4]   2> 1558016 T11066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558016 T11009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558016 T11082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558016 T11034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558016 T11016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558016 T11050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558214 T11085 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1558215 T11085 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:38904_
   [junit4]   2> 1558216 T11085 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:40340_
   [junit4]   2> 1558216 T11085 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:58828_
   [junit4]   2> 1558219 T11085 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:18175_
   [junit4]   2> 1558376 T11070 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard1&name=solrj_collection_shard1_replica1&action=CREATE&numShards=2&wt=javabin&collection=solrj_collection&collection.configName=conf1&version=2&qt=/admin/cores
   [junit4]   2> 1558376 T10999 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard1&name=solrj_collection_shard1_replica2&action=CREATE&numShards=2&wt=javabin&collection=solrj_collection&collection.configName=conf1&version=2&qt=/admin/cores
   [junit4]   2> 1558376 T11040 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard2&name=solrj_collection_shard2_replica2&action=CREATE&numShards=2&wt=javabin&collection=solrj_collection&collection.configName=conf1&version=2&qt=/admin/cores
   [junit4]   2> 1558376 T11054 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard2&name=solrj_collection_shard2_replica1&action=CREATE&numShards=2&wt=javabin&collection=solrj_collection&collection.configName=conf1&version=2&qt=/admin/cores
   [junit4]   2> 1558379 T11070 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1558380 T11070 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1558380 T11009 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1558381 T11054 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1558381 T10999 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1558381 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38904",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:38904_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1558382 T11054 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1558381 T11040 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1558382 T11010 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38904",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:38904_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1558382 T10999 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1558383 T11040 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1558384 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58828",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:58828_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1558385 T11010 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58828",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:58828_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1558387 T11009 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1558387 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40340",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:40340_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1558388 T11010 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40340",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:40340_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1558390 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:18175",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:18175_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 1558390 T11010 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:18175",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:18175_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1558392 T11009 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1558393 T11082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558393 T11066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558393 T11050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558393 T11016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558393 T11009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1558393 T11034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1559381 T11070 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1559382 T11070 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1559382 T11054 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1559383 T11070 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1559383 T11054 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1559383 T11070 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1559384 T11054 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1559383 T10999 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1559384 T11070 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1559384 T11054 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1559385 T11070 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-006/solrj_collection_shard1_replica1/'
   [junit4]   2> 1559384 T11040 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1559384 T10999 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1559386 T11040 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1559386 T10999 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1559386 T11040 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1559385 T11054 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1559387 T11040 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1559387 T10999 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1559387 T11054 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-005/solrj_collection_shard2_replica1/'
   [junit4]   2> 1559388 T11040 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1559388 T10999 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1559388 T11040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-004/solrj_collection_shard2_replica2/'
   [junit4]   2> 1559389 T10999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solrj_collection_shard1_replica2/'
   [junit4]   2> 1559408 T11070 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1559410 T11070 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1559412 T11054 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1559415 T11054 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1559417 T10999 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1559419 T10999 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1559422 T11040 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1559424 T11040 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1559449 T11070 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1559450 T11054 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1559455 T10999 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1559474 T11040 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1559520 T11070 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1559521 T11054 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1559525 T10999 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1559526 T10999 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1559529 T11040 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1559536 T10999 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1559621 T11070 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1559622 T11054 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1559641 T11040 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1559645 T11070 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1559647 T11054 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1559652 T11040 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1559984 T10999 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1559996 T10999 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1560000 T10999 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1560010 T10999 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1560015 T10999 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1560020 T10999 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1560021 T10999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1560023 T10999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1560023 T10999 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1560024 T10999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1560025 T10999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1560025 T10999 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1560025 T10999 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1560026 T10999 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solrj_collection_shard1_replica2/data/
   [junit4]   2> 1560026 T10999 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c82a092
   [junit4]   2> 1560027 T10999 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solrj_collection_shard1_replica2/data
   [junit4]   2> 1560028 T10999 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 1560028 T10999 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1560029 T10999 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-002/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 1560029 T10999 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=18, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1560030 T10999 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@473da497 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2000da0b),segFN=segments_1,generation=1}
   [junit4]   2> 1560031 T10999 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1560044 T10999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1560045 T10999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1560045 T10999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1560045 T10999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1560046 T10999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1560046 T10999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1560047 T10999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1560047 T10999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1560047 T10999 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1560048 T10999 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1560049 T10999 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1560049 T10999 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1560050 T10999 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1560051 T10999 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1560051 T10999 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1560052 T10999 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1560052 T10999 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1560053 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1560054 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1560055 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1560055 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1560056 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1560056 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1560057 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1560058 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1560058 T10999 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1560059 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1560059 T10999 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1560083 T10999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1560086 T10999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1560088 T10999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1560090 T10999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1560098 T10999 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1560099 T10999 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1560100 T10999 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=0.8427734375, floorSegmentMB=1.33203125, forceMergeDeletesPctAllowed=27.110828995221215, segmentsPerTier=50.0, maxCFSSegmentSizeMB=0.2392578125, noCFSRatio=0.16684406884690528
   [junit4]   2> 1560101 T10999 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@473da497 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2000da0b),segFN=segments_1,generation=1}
   [junit4]   2> 1560101 T10999 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1560101 T10999 oass.SolrIndexSearcher.<init> Opening Searcher@5f12ccdf[solrj_collection_shard1_replica2] main
   [junit4]   2> 1560102 T10999 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1560103 T10999 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1560103 T10999 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1560103 T10999 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1560104 T10999 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1560104 T10999 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1560105 T11070 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1560105 T10999 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1560105 T10999 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1560106 T10999 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1560107 T11054 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1560108 T10999 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1560108 T10999 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1560109 T10999 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1560109 T10999 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1560110 T11094 oasc.SolrCore.registerSearcher [solrj_collection_shard1_replica2] Registered new searcher Searcher@5f12ccdf[solrj_collection_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1560110 T10999 oasc.CoreContainer.registerCore registering core: solrj_collection_shard1_replica2
   [junit4]   2> 1560111 T10999 oasc.ZkController.register Register replica - core:solrj_collection_shard1_replica2 address:https://127.0.0.1:40340 collection:solrj_collection shard:shard1
   [junit4]   2> 1560111 T11070 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1560112 T10999 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leader_elect/shard1/election
   [junit4]   2> 1560114 T11054 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1560115 T11070 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1560115 T11040 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1560116 T10999 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1560117 T11054 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1560117 T11009 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1560118 T10999 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=0ms
   [junit4]   2> 1560118 T10999 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Was waiting for replicas to come up, but they are taking too long - assuming they won't come back till later
   [junit4]   2> 1560118 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection"} current state version: 3
   [junit4]   2> 1560118 T10999 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2519 name=solrj_collection_shard1_replica2 org.apache.solr.core.SolrCore@794a1ca0 url=https://127.0.0.1:40340/solrj_collection_shard1_replica2 node=127.0.0.1:40340_ C2519_STATE=coll:solrj_collection core:solrj_collection_shard1_replica2 props:{core=solrj_collection_shard1_replica2, base_url=https://127.0.0.1:40340, node_name=127.0.0.1:40340_, state=down}
   [junit4]   2> 1560119 T10999 C2519 P40340 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:40340/solrj_collection_shard1_replica2/
   [junit4]   2> 1560119 T10999 C2519 P40340 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica2 url=https://127.0.0.1:40340 START replicas=[https://127.0.0.1:38904/solrj_collection_shard1_replica1/] nUpdates=100
   [junit4]   2> 1560120 T11066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1560120 T10999 C2519 P40340 oasu.PeerSync.sync PeerSync: core=solrj_collection_shard1_replica2 url=https://127.0.0.1:40340 DONE.  We have no versions.  sync failed.
   [junit4]   2> 1560121 T10999 C2519 P40340 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1560120 T11034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1560122 T11040 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1560120 T11082 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1560120 T11016 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1560120 T11050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1560120 T11009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1560122 T10999 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1560124 T10999 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:40340/solrj_collection_shard1_replica2/ shard1
   [junit4]   2> 1560124 T10999 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection/leaders/shard1
   [junit4]   2> 1560127 T11040 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1560127 T11070 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1560128 T11097 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1560129 T11010 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:40340",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 1560130 T11054 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1560132 T11070 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1560134 T11054 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1560136 T11070 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1560137 T11070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1560138 T11070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1560138 T11054 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1560138 T11070 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1560139 T11040 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1560139 T11054 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1560139 T11070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1560139 T11054 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1560140 T11070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1560140 T11054 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1560140 T11070 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 1560141 T11070 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1560141 T11054 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1560141 T11070 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-006/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-AFE8A0D51025B96-001/tempDir-006/solrj_collection_shard1_replica1/data/
   [junit4]   2> 1560142 T11054 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRIN

[...truncated too long message...]

r-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-AFE8A0D51025B96-001/tempDir-001/jetty9;done=false>>]
   [junit4]   2> 2675583 T6309 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-AFE8A0D51025B96-001/tempDir-001/jetty9
   [junit4]   2> 2675584 T6309 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-AFE8A0D51025B96-001/tempDir-001/jetty9/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-AFE8A0D51025B96-001/tempDir-001/jetty9/index;done=false>>]
   [junit4]   2> 2675584 T6309 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-AFE8A0D51025B96-001/tempDir-001/jetty9/index
   [junit4]   2> 2675585 T6309 oasc.Overseer.close Overseer (id=92991328917520407-127.0.0.1:27852_-n_0000000009) closing
   [junit4]   2> 2675585 T6910 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:27852_
   [junit4]   2> 2677086 T6909 oascc.ZkStateReader$7.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2677087 T6879 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2677089 T6309 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 2682092 T6309 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 2682095 T6309 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:21237 21237
   [junit4]   2> 2682303 T6310 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:21237 21237
   [junit4]   2> 2682304 T6310 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		14	/solr/clusterstate.json
   [junit4]   2> 		14	/solr/aliases.json
   [junit4]   2> 		13	/solr/configs/conf1
   [junit4]   2> 		12	/solr/collections/collection1/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent data watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		4	/solr/overseer_elect/election/92991328917520387-127.0.0.1:25714_-n_0000000000
   [junit4]   2> 		3	/solr/overseer_elect/election/92991328917520407-127.0.0.1:27852_-n_0000000009
   [junit4]   2> 		2	/solr/overseer_elect/election/92991328917520399-127.0.0.1:25732_-n_0000000005
   [junit4]   2> 		2	/solr/collections/collection1/leader_elect/shard1/election/92991328917520395-core_node3-n_0000000000
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		11	/solr/live_nodes
   [junit4]   2> 		6	/solr/overseer/collection-queue-work
   [junit4]   2> 		6	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=AFE8A0D51025B96 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=vi_VN -Dtests.timezone=Etc/GMT+11 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE  319s J1 | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: document count mismatch.  control=1502 sum(shards)=1503 cloudClient=1503
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AFE8A0D51025B96:8B180415265D3BAA]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1308)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:232)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2682320 T6309 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-AFE8A0D51025B96-001
   [junit4]   2> 319083 T6308 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {rnd_b=PostingsFormat(name=Memory doPackFST= false), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=PostingsFormat(name=Memory doPackFST= false), _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_i=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, sim=DefaultSimilarity, locale=vi_VN, timezone=Etc/GMT+11
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=82851432,total=437780480
   [junit4]   2> NOTE: All tests run in this JVM: [CoreAdminCreateDiscoverTest, ExitableDirectoryReaderTest, DateFieldTest, TestCollapseQParserPlugin, TestOmitPositions, OpenExchangeRatesOrgProviderTest, AddBlockUpdateTest, LoggingHandlerTest, MigrateRouteKeyTest, AsyncMigrateRouteKeyTest, CursorMarkTest, TestBM25SimilarityFactory, IndexSchemaTest, SuggestComponentTest, TestSolr4Spatial, DistributedFacetPivotWhiteBoxTest, TestShortCircuitedRequests, OpenCloseCoreStressTest, TestBlendedInfixSuggestions, ChaosMonkeySafeLeaderTest, InfoHandlerTest, TestDocSet, TestBadConfig, TermVectorComponentTest, TestExceedMaxTermLength, DOMUtilTest, TestInitParams, TestFieldResource, SuggesterFSTTest, FieldMutatingUpdateProcessorTest, TestDocBasedVersionConstraints, TestNoOpRegenerator, BasicZkTest, TestReplicaProperties, ChangedSchemaMergeTest, TestRecovery, ResourceLoaderTest, CoreContainerCoreInitFailuresTest, ModifyConfFileTest, TestQuerySenderNoQuery, QueryElevationComponentTest, TestQuerySenderListener, TestSchemaVersionResource, BlockDirectoryTest, DistributedFacetPivotSmallTest, HdfsBasicDistributedZkTest, FieldAnalysisRequestHandlerTest, ExternalCollectionsTest, ZkControllerTest, TestStressVersions, HdfsWriteToMultipleCollectionsTest, FullSolrCloudDistribCmdsTest, SpellPossibilityIteratorTest, TestPerFieldSimilarity, SolrInfoMBeanTest, LegacyHTMLStripCharFilterTest, TestZkChroot, TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, AutoCommitTest, TestDistributedGrouping, TestReload, DistributedTermsComponentTest, SolrCmdDistributorTest, BadIndexSchemaTest, TestSort, TestFunctionQuery, OverseerCollectionProcessorTest, TestIndexSearcher, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, DocValuesTest, SuggesterTest, SuggesterWFSTTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, DefaultValueUpdateProcessorTest, TestRemoteStreaming, XsltUpdateRequestHandlerTest, LukeRequestHandlerTest, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, TestQueryTypes, TermsComponentTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, TestCollationField, TestCSVResponseWriter, UniqFieldsUpdateProcessorFactoryTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, TestComponentsName, TestBinaryResponseWriter, SOLR749Test, TestSolrIndexConfig, ResponseLogComponentTest, SolrIndexConfigTest, TestSolrDeletionPolicy2, MultiTermTest, TestBinaryField, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, TestSweetSpotSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestDefaultSimilarityFactory, ScriptEngineTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J1 in 319.62s, 1 test, 1 failure <<< FAILURES!

[...truncated 409 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:190: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:970: There were test failures: 452 suites, 1849 tests, 2 failures, 57 ignored (10 assumptions)

Total time: 172 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #705
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 698983091 bytes
Compression is 0.0%
Took 2 min 5 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message