lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0_05) - Build # 10684 - Still Failing!
Date Sun, 13 Jul 2014 10:56:05 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10684/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard3 is not consistent.  Got 159 from https://127.0.0.1:43188/collection1lastClient and got 135 from https://127.0.0.1:37995/collection1

Stack Trace:
java.lang.AssertionError: shard3 is not consistent.  Got 159 from https://127.0.0.1:43188/collection1lastClient and got 135 from https://127.0.0.1:37995/collection1
	at __randomizedtesting.SeedInfo.seed([788BAFCA394B56F3:F96D21D24E1436CF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.RollingRestartTest.testDistribSearch

Error Message:
No overseer designate as leader found after restart #1: 127.0.0.1:38817__

Stack Trace:
java.lang.AssertionError: No overseer designate as leader found after restart #1: 127.0.0.1:38817__
	at __randomizedtesting.SeedInfo.seed([788BAFCA394B56F3:F96D21D24E1436CF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:100)
	at org.apache.solr.cloud.RollingRestartTest.doTest(RollingRestartTest.java:61)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11410 lines...]
   [junit4] Suite: org.apache.solr.cloud.RollingRestartTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/init-core-data-001
   [junit4]   2> 445819 T2706 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 445819 T2706 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_/
   [junit4]   2> 445821 T2706 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 445822 T2706 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 445822 T2707 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 445922 T2706 oasc.ZkTestServer.run start zk server on port:38707
   [junit4]   2> 445923 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 445925 T2713 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77db09b7 name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 445925 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 445926 T2706 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 445929 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 445930 T2715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e42d1ed name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 445930 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 445930 T2706 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 445931 T2706 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 445932 T2706 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 445933 T2706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 445934 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 445934 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 445936 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 445936 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 445937 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 445938 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 445939 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 445939 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 445940 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 445941 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 445942 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 445942 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 445943 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 445943 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 445944 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 445945 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 445946 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 445946 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 445947 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 445948 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 445949 T2706 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 445949 T2706 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 445951 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 445952 T2717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@503237a0 name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 445952 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446187 T2706 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 446190 T2706 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 446199 T2706 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:43392
   [junit4]   2> 446201 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 446201 T2706 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 446202 T2706 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002
   [junit4]   2> 446202 T2706 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/'
   [junit4]   2> 446217 T2706 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/solr.xml
   [junit4]   2> 446226 T2706 oasc.CoreContainer.<init> New CoreContainer 277018058
   [junit4]   2> 446227 T2706 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/]
   [junit4]   2> 446227 T2706 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 446228 T2706 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 446228 T2706 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 446228 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 446229 T2706 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 446229 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 446229 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 446230 T2706 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 446230 T2706 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 446232 T2706 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 446233 T2706 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 446233 T2706 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 446234 T2706 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38707/solr
   [junit4]   2> 446234 T2706 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 446235 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446236 T2728 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ba83ecb name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446236 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446238 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446239 T2730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48c70c4d name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446240 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446241 T2706 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 446243 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 446248 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 446250 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 446252 T2706 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 446254 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 446256 T2706 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43392__
   [junit4]   2> 446256 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43392__
   [junit4]   2> 446258 T2706 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 446260 T2706 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 446262 T2706 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43392__
   [junit4]   2> 446262 T2706 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 446263 T2706 oasc.Overseer.start Overseer (id=92094321097703428-127.0.0.1:43392__-n_0000000000) starting
   [junit4]   2> 446265 T2706 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 446270 T2732 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 446271 T2706 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 446272 T2706 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 446273 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 446275 T2731 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 446276 T2733 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 446276 T2733 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 446277 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 446277 T2733 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 446278 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 446278 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43392/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43392__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 446278 T2731 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 446278 T2731 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 446279 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 446280 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447277 T2733 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 447278 T2733 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 447278 T2733 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 447279 T2733 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 447279 T2733 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 447280 T2733 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/'
   [junit4]   2> 447280 T2733 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 447280 T2733 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 447281 T2733 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 447297 T2733 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 447313 T2733 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 447314 T2733 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 447318 T2733 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 447429 T2733 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 447430 T2733 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 447430 T2733 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 447434 T2733 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 447435 T2733 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 447442 T2733 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 447444 T2733 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 447446 T2733 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 447447 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 447447 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 447447 T2733 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 447448 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 447448 T2733 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 447448 T2733 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 447448 T2733 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 447448 T2733 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-002/collection1/, dataDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/
   [junit4]   2> 447449 T2733 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@200cabd5
   [junit4]   2> 447449 T2733 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data
   [junit4]   2> 447449 T2733 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index/
   [junit4]   2> 447449 T2733 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 447450 T2733 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index
   [junit4]   2> 447450 T2733 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=34.9697265625, floorSegmentMB=1.3818359375, forceMergeDeletesPctAllowed=7.079538686834988, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21477591279373576
   [junit4]   2> 447520 T2733 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 447520 T2733 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 447522 T2733 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 447523 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 447523 T2733 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 447523 T2733 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 447523 T2733 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 447523 T2733 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 447524 T2733 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 447525 T2733 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 447526 T2733 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 447527 T2733 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 447527 T2733 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 447527 T2733 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=61.740234375, floorSegmentMB=1.59765625, forceMergeDeletesPctAllowed=7.764287375918466, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.7275390625, noCFSRatio=1.0
   [junit4]   2> 447528 T2733 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 447528 T2733 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 447528 T2733 oass.SolrIndexSearcher.<init> Opening Searcher@31c6243[collection1] main
   [junit4]   2> 447529 T2733 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 447530 T2733 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 447530 T2733 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 447530 T2733 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 447530 T2733 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 447531 T2733 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 447531 T2733 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 447531 T2733 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 447531 T2733 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 447531 T2733 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 447533 T2734 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31c6243[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 447533 T2733 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 447534 T2737 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:43392/_ collection:control_collection shard:shard1
   [junit4]   2> 447534 T2706 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 447535 T2737 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 447535 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 447537 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447537 T2737 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 447538 T2739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6620eed3 name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447538 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447538 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447539 T2737 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 447539 T2737 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 447539 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 447539 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C894 name=collection1 org.apache.solr.core.SolrCore@3ab1d5d4 url=https://127.0.0.1:43392/_/collection1 node=127.0.0.1:43392__ C894_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:43392/_, core=collection1, node_name=127.0.0.1:43392__}
   [junit4]   2> 447539 T2737 C894 P43392 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:43392/_/collection1/
   [junit4]   2> 447540 T2737 C894 P43392 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 447540 T2737 C894 P43392 oasc.SyncStrategy.syncToMe https://127.0.0.1:43392/_/collection1/ has no replicas
   [junit4]   2> 447540 T2737 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:43392/_/collection1/ shard1
   [junit4]   2> 447540 T2737 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 447540 T2706 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 447541 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447543 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447544 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447645 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447646 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447694 T2737 oasc.ZkController.register We are https://127.0.0.1:43392/_/collection1/ and leader is https://127.0.0.1:43392/_/collection1/
   [junit4]   2> 447694 T2737 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:43392/_
   [junit4]   2> 447694 T2737 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 447694 T2737 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 447694 T2737 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 447695 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447695 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447695 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447696 T2737 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 447696 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 447697 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43392/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43392__",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 447698 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 447793 T2706 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 447794 T2706 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 447796 T2706 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 447797 T2706 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:45364
   [junit4]   2> 447799 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 447799 T2706 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 447799 T2706 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003
   [junit4]   2> 447799 T2706 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/'
   [junit4]   2> 447799 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447799 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 447809 T2706 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/solr.xml
   [junit4]   2> 447825 T2706 oasc.CoreContainer.<init> New CoreContainer 429451
   [junit4]   2> 447826 T2706 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/]
   [junit4]   2> 447826 T2706 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 447826 T2706 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 447826 T2706 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 447827 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 447828 T2706 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 447828 T2706 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 447829 T2706 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 447830 T2706 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 447830 T2706 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 447830 T2706 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38707/solr
   [junit4]   2> 447830 T2706 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 447831 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447832 T2750 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e891c7c name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447832 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447834 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447834 T2752 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c8afb5c name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447835 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447838 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 448839 T2706 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45364__
   [junit4]   2> 448840 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45364__
   [junit4]   2> 448842 T2739 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 448842 T2730 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 448842 T2752 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 448845 T2753 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 448845 T2753 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 448846 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448846 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448846 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448846 T2753 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 448847 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 448847 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45364/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45364__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 448847 T2731 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 448847 T2731 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 448848 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 448949 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 448949 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 448949 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 449846 T2753 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 449847 T2753 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 449847 T2753 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 449847 T2753 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 449848 T2753 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 449848 T2753 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/'
   [junit4]   2> 449849 T2753 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 449849 T2753 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 449850 T2753 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 449869 T2753 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 449878 T2753 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 449879 T2753 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 449883 T2753 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 449997 T2753 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 449997 T2753 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 449997 T2753 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 450001 T2753 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 450002 T2753 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 450016 T2753 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 450017 T2753 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 450019 T2753 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 450020 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 450020 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 450020 T2753 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 450024 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 450024 T2753 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 450024 T2753 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 450024 T2753 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 450025 T2753 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-003/collection1/, dataDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/
   [junit4]   2> 450025 T2753 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@200cabd5
   [junit4]   2> 450025 T2753 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1
   [junit4]   2> 450026 T2753 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index/
   [junit4]   2> 450026 T2753 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 450026 T2753 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index
   [junit4]   2> 450027 T2753 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=34.9697265625, floorSegmentMB=1.3818359375, forceMergeDeletesPctAllowed=7.079538686834988, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21477591279373576
   [junit4]   2> 450037 T2753 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 450037 T2753 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 450039 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 450040 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 450041 T2753 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 450041 T2753 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 450041 T2753 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 450041 T2753 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 450042 T2753 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 450043 T2753 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 450043 T2753 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 450044 T2753 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 450046 T2753 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 450046 T2753 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 450047 T2753 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=61.740234375, floorSegmentMB=1.59765625, forceMergeDeletesPctAllowed=7.764287375918466, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.7275390625, noCFSRatio=1.0
   [junit4]   2> 450048 T2753 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 450048 T2753 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 450048 T2753 oass.SolrIndexSearcher.<init> Opening Searcher@31e15422[collection1] main
   [junit4]   2> 450049 T2753 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 450049 T2753 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 450050 T2753 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 450050 T2753 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 450050 T2753 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 450050 T2753 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 450051 T2753 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 450051 T2753 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 450051 T2753 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 450051 T2753 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 450053 T2754 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31e15422[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 450055 T2753 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 450055 T2757 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:45364/_ collection:collection1 shard:shard2
   [junit4]   2> 450056 T2706 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 450056 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 450056 T2757 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 450059 T2757 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 450072 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450072 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450072 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450073 T2757 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 450073 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 450074 T2757 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 450076 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C895 name=collection1 org.apache.solr.core.SolrCore@4e296a86 url=https://127.0.0.1:45364/_/collection1 node=127.0.0.1:45364__ C895_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:45364/_, core=collection1, node_name=127.0.0.1:45364__}
   [junit4]   2> 450076 T2757 C895 P45364 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:45364/_/collection1/
   [junit4]   2> 450076 T2757 C895 P45364 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 450077 T2757 C895 P45364 oasc.SyncStrategy.syncToMe https://127.0.0.1:45364/_/collection1/ has no replicas
   [junit4]   2> 450077 T2757 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:45364/_/collection1/ shard2
   [junit4]   2> 450077 T2757 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 450081 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450084 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450185 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450185 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450185 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450232 T2757 oasc.ZkController.register We are https://127.0.0.1:45364/_/collection1/ and leader is https://127.0.0.1:45364/_/collection1/
   [junit4]   2> 450233 T2757 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:45364/_
   [junit4]   2> 450233 T2757 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 450233 T2757 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 450233 T2757 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 450235 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450235 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450235 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450235 T2757 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 450235 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 450236 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45364/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45364__",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 450238 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 450328 T2706 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 450328 T2706 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 450331 T2706 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 450332 T2706 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38817
   [junit4]   2> 450334 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 450334 T2706 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 450334 T2706 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004
   [junit4]   2> 450335 T2706 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/'
   [junit4]   2> 450339 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450339 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450339 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 450346 T2706 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/solr.xml
   [junit4]   2> 450354 T2706 oasc.CoreContainer.<init> New CoreContainer 1129178407
   [junit4]   2> 450354 T2706 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/]
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 450355 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 450356 T2706 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 450358 T2706 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 450358 T2706 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 450359 T2706 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 450359 T2706 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38707/solr
   [junit4]   2> 450359 T2706 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 450360 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 450361 T2768 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@770558ae name:ZooKeeperConnection Watcher:127.0.0.1:38707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 450361 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 450362 T2706 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 450363 T2770 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26e0013d name:ZooKeeperConnection Watcher:127.0.0.1:38707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 450363 T2706 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 450366 T2706 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 451368 T2706 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38817__
   [junit4]   2> 451369 T2706 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38817__
   [junit4]   2> 451371 T2739 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451371 T2730 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451371 T2752 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451371 T2770 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 451375 T2771 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 451375 T2771 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 451376 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451376 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451376 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451376 T2771 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451376 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 451377 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38817/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38817__",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451377 T2731 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 451377 T2731 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 451378 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451479 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 451479 T2770 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 451479 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 451479 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452376 T2771 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 452376 T2771 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 452377 T2771 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452377 T2771 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 452378 T2771 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 452378 T2771 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/'
   [junit4]   2> 452378 T2771 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 452379 T2771 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 452379 T2771 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 452407 T2771 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
   [junit4]   2> 452416 T2771 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 452418 T2771 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 452422 T2771 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 452540 T2771 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 452540 T2771 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 452541 T2771 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 452544 T2771 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 452546 T2771 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 452553 T2771 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 452555 T2771 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 452558 T2771 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 452558 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 452559 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 452559 T2771 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 452560 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 452560 T2771 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 452560 T2771 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 452560 T2771 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 452561 T2771 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-004/collection1/, dataDir=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/
   [junit4]   2> 452561 T2771 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@200cabd5
   [junit4]   2> 452561 T2771 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2
   [junit4]   2> 452562 T2771 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index/
   [junit4]   2> 452562 T2771 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 452563 T2771 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index
   [junit4]   2> 452563 T2771 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=34.9697265625, floorSegmentMB=1.3818359375, forceMergeDeletesPctAllowed=7.079538686834988, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21477591279373576
   [junit4]   2> 452571 T2771 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 452571 T2771 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 452574 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 452575 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 452575 T2771 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 452575 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 452576 T2771 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 452577 T2771 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 452577 T2771 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 452578 T2771 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 452578 T2771 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 452578 T2771 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 452579 T2771 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 452579 T2771 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 452579 T2771 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 452582 T2771 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 452583 T2771 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 452583 T2771 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 452584 T2771 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=61.740234375, floorSegmentMB=1.59765625, forceMergeDeletesPctAllowed=7.764287375918466, segmentsPerTier=13.0, maxCFSSegmentSizeMB=1.7275390625, noCFSRatio=1.0
   [junit4]   2> 452585 T2771 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.RollingRestartTest-788BAFCA394B56F3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 452585 T2771 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 452586 T2771 oass.SolrIndexSearcher.<init> Opening Searcher@40e852a5[collection1] main
   [junit4]   2> 452586 T2771 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 452587 T2771 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 452587 T2771 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 452587 T2771 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 452588 T2771 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 452588 T2771 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 452588 T2771 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 452589 T2771 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 452589 T2771 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 452589 T2771 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 452591 T2772 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@40e852a5[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 452593 T2771 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 452594 T2775 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38817/_ collection:collection1 shard:shard1
   [junit4]   2> 452594 T2706 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 452594 T2706 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 452595 T2775 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 452598 T2775 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 452598 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452599 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452599 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452599 T2775 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 452599 T2775 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 452599 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C896 name=collection1 org.apache.solr.core.SolrCore@3ee53a83 url=https://127.0.0.1:38817/_/collection1 node=127.0.0.1:38817__ C896_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:38817/_, core=collection1, node_name=127.0.0.1:38817__}
   [junit4]   2> 452600 T2775 C896 P38817 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38817/_/collection1/
   [junit4]   2> 452600 T2775 C896 P38817 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 452600 T2775 C896 P38817 oasc.SyncStrategy.syncToMe https://127.0.0.1:38817/_/collection1/ has no replicas
   [junit4]   2> 452600 T2775 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38817/_/collection1/ shard1
   [junit4]   2> 452600 T2775 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 452601 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452603 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452604 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452705 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452706 T2730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452706 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452705 T2770 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452753 T2775 oasc.ZkController.register We are https://127.0.0.1:38817/_/collection1/ and leader is https://127.0.0.1:38817/_/collection1/
   [junit4]   2> 452754 T2775 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38817/_
   [junit4]   2> 452754 T2775 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 452754 T2775 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 452754 T2775 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 452755 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452755 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452755 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452755 T2775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 452756 T2731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 452756 T2731 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38817/_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38817__",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 452757 T2730 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452858 T2739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452859 T2752 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 452859 T2730 oascc.ZkStateReader$2.process A 

[...truncated too long message...]

jetty9
   [junit4]   2> 739816 T3579 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 739817 T3579 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 740063 T3654 C2657 P42194 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 740064 T3654 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 740064 T3654 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@63da0c51
   [junit4]   2> 740065 T3654 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=79,cumulative_deletesById=52,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=7948,transaction_logs_total_number=2}
   [junit4]   2> 740066 T3654 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 740066 T3654 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 740067 T3654 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 740068 T3654 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 740069 T3654 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 740069 T3654 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12;done=false>>]
   [junit4]   2> 740070 T3654 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12
   [junit4]   2> 740070 T3654 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12/index.20140713104136674 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12/index.20140713104136674;done=false>>]
   [junit4]   2> 740070 T3654 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001/tempDir-001/jetty12/index.20140713104136674
   [junit4]   2> 740071 T3654 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 740071 T3654 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-788BAFCA394B56F3-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene49), sim=DefaultSimilarity, locale=sr_ME, timezone=Africa/Nouakchott
   [junit4]   2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=66964352,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestReloadAndDeleteDocs, TestIBSimilarityFactory, DocExpirationUpdateProcessorFactoryTest, TestExtendedDismaxParser, DistributedDebugComponentTest, TestFunctionQuery, TestRandomFaceting, TestPostingsSolrHighlighter, TestCoreContainer, SolrRequestParserTest, TestSuggestSpellingConverter, OverseerStatusTest, NumericFieldsTest, MultiTermTest, TestBadConfig, DistribCursorPagingTest, TestXIncludeConfig, HdfsWriteToMultipleCollectionsTest, TestClassNameShortening, TestDistributedGrouping, TestLRUCache, CurrencyFieldXmlFileTest, DOMUtilTest, DeleteReplicaTest, TestOverriddenPrefixQueryForCustomFieldType, InfoHandlerTest, DirectSolrConnectionTest, TestFreeTextSuggestions, LegacyHTMLStripCharFilterTest, TestDynamicFieldResource, TestQuerySenderNoQuery, QueryParsingTest, EnumFieldTest, TestSweetSpotSimilarityFactory, TestSolrQueryParserDefaultOperatorResource, RequestHandlersTest, DeleteInactiveReplicaTest, TestStandardQParsers, DistributedTermsComponentTest, SchemaVersionSpecificBehaviorTest, TestRecovery, TestLazyCores, AutoCommitTest, MigrateRouteKeyTest, PreAnalyzedFieldTest, CopyFieldTest, SolrInfoMBeanTest, ResponseHeaderTest, LoggingHandlerTest, SolrIndexConfigTest, TestExpandComponent, TestIndexSearcher, TestRandomMergePolicy, TestReversedWildcardFilterFactory, TestNonNRTOpen, AliasIntegrationTest, SolrCoreTest, DistributedSuggestComponentTest, TestExceedMaxTermLength, BasicDistributedZk2Test, TestGroupingSearch, TestRealTimeGet, PolyFieldTest, RankQueryTest, TestRTGBase, CacheHeaderTest, DistributedQueryElevationComponentTest, TestSolrXmlPersistence, TestTolerantSearch, TestFieldTypeResource, TestMergePolicyConfig, TestNonDefinedSimilarityFactory, TestCollationField, HardAutoCommitTest, TestSolrQueryParserResource, SimplePostToolTest, SampleTest, TestDocBasedVersionConstraints, TestCSVLoader, IndexSchemaRuntimeFieldTest, TestOmitPositions, TestWriterPerf, TestFiltering, TestZkChroot, TestManagedResource, SearchHandlerTest, TestPseudoReturnFields, UpdateRequestProcessorFactoryTest, SolrPluginUtilsTest, FileBasedSpellCheckerTest, TestMissingGroups, BlockCacheTest, TestIndexingPerformance, DistributedQueryComponentCustomSortTest, TestLFUCache, ConvertedLegacyTest, TestBinaryField, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed on J1 in 89.70s, 1 test, 1 failure <<< FAILURES!

[...truncated 697 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1299: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:923: There were test failures: 407 suites, 1660 tests, 2 failures, 37 ignored (16 assumptions)

Total time: 70 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message