lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4616 - Failure
Date Sat, 15 Feb 2014 04:34:55 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4616/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:22193/collection1, https://127.0.0.1:28110/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:22193/collection1, https://127.0.0.1:28110/collection1]
	at __randomizedtesting.SeedInfo.seed([8850D5789F557AF7:9B65B60E80A1ACB]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:362)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1356)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:290)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:107)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: no servers hosting shard: 
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:293)
	... 50 more


REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
There were too many update fails - we expect it can happen, but shouldn't easily

Stack Trace:
java.lang.AssertionError: There were too many update fails - we expect it can happen, but shouldn't easily
	at __randomizedtesting.SeedInfo.seed([8850D5789F557AF7:9B65B60E80A1ACB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:207)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 10334 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 667584 T7426 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 667585 T7426 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 667593 T7426 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1392436426890
   [junit4]   2> 667595 T7426 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 667596 T7427 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 667697 T7426 oasc.ZkTestServer.run start zk server on port:28338
   [junit4]   2> 667698 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 667704 T7433 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@212487ba name:ZooKeeperConnection Watcher:127.0.0.1:28338 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 667704 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 667705 T7426 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 667783 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 667882 T7435 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79cadc72 name:ZooKeeperConnection Watcher:127.0.0.1:28338/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 667882 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 667883 T7426 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 667930 T7426 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 667933 T7426 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 667966 T7426 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 667974 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 667975 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 668009 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 668010 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 668119 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 668120 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 668144 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 668145 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 668152 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 668153 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 668215 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 668216 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 668225 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 668226 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 668336 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 668337 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 668344 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 668345 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 668352 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 668352 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 668374 T7426 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 668375 T7426 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 668702 T7426 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 668707 T7426 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:32752
   [junit4]   2> 668708 T7426 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 668709 T7426 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 668709 T7426 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685
   [junit4]   2> 668710 T7426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/'
   [junit4]   2> 668750 T7426 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/solr.xml
   [junit4]   2> 668851 T7426 oasc.CoreContainer.<init> New CoreContainer 1104127102
   [junit4]   2> 668852 T7426 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/]
   [junit4]   2> 668854 T7426 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 668854 T7426 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 668855 T7426 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 668855 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 668856 T7426 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 668856 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 668856 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 668857 T7426 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 668857 T7426 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 668870 T7426 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 668871 T7426 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 668871 T7426 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 668872 T7426 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28338/solr
   [junit4]   2> 668872 T7426 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 668873 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 668879 T7446 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27386df3 name:ZooKeeperConnection Watcher:127.0.0.1:28338 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 668880 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 668937 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 668945 T7448 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fb616d3 name:ZooKeeperConnection Watcher:127.0.0.1:28338/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 668945 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 668985 T7426 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 669068 T7426 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 669072 T7426 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 669080 T7426 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32752_
   [junit4]   2> 669176 T7426 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32752_
   [junit4]   2> 669184 T7426 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 669191 T7426 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 669227 T7426 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:32752_
   [junit4]   2> 669227 T7426 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 669238 T7426 oasc.Overseer.start Overseer (id=91254713673252867-127.0.0.1:32752_-n_0000000000) starting
   [junit4]   2> 669259 T7426 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 669290 T7450 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 669291 T7426 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 669291 T7450 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 669335 T7426 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 669337 T7426 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 669339 T7449 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 669346 T7451 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 669346 T7451 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 669357 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 669357 T7451 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 669358 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 669359 T7449 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32752",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32752_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 669359 T7449 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 669360 T7449 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 669376 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 669404 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 670358 T7451 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 670358 T7451 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/collection1
   [junit4]   2> 670359 T7451 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 670360 T7451 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 670360 T7451 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 670361 T7451 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 670361 T7451 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/collection1/'
   [junit4]   2> 670363 T7451 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/collection1/lib/README' to classloader
   [junit4]   2> 670363 T7451 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/collection1/lib/classes/' to classloader
   [junit4]   2> 670431 T7451 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 670505 T7451 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 670607 T7451 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 670618 T7451 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 671238 T7451 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 671242 T7451 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 671245 T7451 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 671250 T7451 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 671276 T7451 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 671325 T7451 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1392436427685/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/control/data/
   [junit4]   2> 671326 T7451 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c4b8217
   [junit4]   2> 671328 T7451 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/control/data
   [junit4]   2> 671329 T7451 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/control/data/index/
   [junit4]   2> 671330 T7451 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 671331 T7451 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/control/data/index
   [junit4]   2> 671332 T7451 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=36.970703125, floorSegmentMB=0.841796875, forceMergeDeletesPctAllowed=16.76668501874187, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21803891689922628
   [junit4]   2> 671410 T7451 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 671410 T7451 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 671416 T7451 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 671416 T7451 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 671416 T7451 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 671417 T7451 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 671417 T7451 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 671417 T7451 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 671418 T7451 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 671419 T7451 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 671419 T7451 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 671420 T7451 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 671420 T7451 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 671420 T7451 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 671421 T7451 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 671422 T7451 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 671422 T7451 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 671423 T7451 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 671423 T7451 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 671440 T7451 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 671445 T7451 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 671445 T7451 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 671446 T7451 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 671448 T7451 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 671448 T7451 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 671449 T7451 oass.SolrIndexSearcher.<init> Opening Searcher@4f260411[collection1] main
   [junit4]   2> 671452 T7452 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4f260411[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 671456 T7451 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 671456 T7426 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
   [junit4]   2> 671457 T7426 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 671456 T7455 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32752 collection:control_collection shard:shard1
   [junit4]   2> 671459 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 671459 T7455 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 671462 T7457 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33ebc1e0 name:ZooKeeperConnection Watcher:127.0.0.1:28338/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 671462 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 671490 T7426 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 671520 T7426 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 671536 T7455 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 671595 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 671595 T7455 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 671596 T7455 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C473 name=collection1 org.apache.solr.core.SolrCore@223c36de url=http://127.0.0.1:32752/collection1 node=127.0.0.1:32752_ C473_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:32752, core=collection1, node_name=127.0.0.1:32752_}
   [junit4]   2> 671596 T7455 C473 P32752 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32752/collection1/
   [junit4]   2> 671596 T7455 C473 P32752 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 671596 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 671597 T7455 C473 P32752 oasc.SyncStrategy.syncToMe http://127.0.0.1:32752/collection1/ has no replicas
   [junit4]   2> 671597 T7455 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32752/collection1/ shard1
   [junit4]   2> 671598 T7455 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 671620 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 671726 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 671726 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 671808 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 671871 T7426 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 671873 T7426 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 671876 T7426 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:47517
   [junit4]   2> 671876 T7426 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 671877 T7426 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 671877 T7426 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817
   [junit4]   2> 671878 T7426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/'
   [junit4]   2> 671917 T7426 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/solr.xml
   [junit4]   2> 671985 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 672014 T7426 oasc.CoreContainer.<init> New CoreContainer 376477418
   [junit4]   2> 672015 T7426 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/]
   [junit4]   2> 672017 T7426 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 672017 T7426 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 672018 T7426 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 672018 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 672019 T7426 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 672019 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 672020 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 672020 T7426 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 672021 T7426 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 672033 T7426 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 672033 T7426 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 672034 T7426 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 672034 T7426 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28338/solr
   [junit4]   2> 672035 T7426 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 672036 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672039 T7468 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39c3e828 name:ZooKeeperConnection Watcher:127.0.0.1:28338 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672039 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672051 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672057 T7470 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fc0c928 name:ZooKeeperConnection Watcher:127.0.0.1:28338/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672057 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672121 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 672121 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 672244 T7426 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 672267 T7455 oasc.ZkController.register We are http://127.0.0.1:32752/collection1/ and leader is http://127.0.0.1:32752/collection1/
   [junit4]   2> 672267 T7455 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32752
   [junit4]   2> 672267 T7455 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 672267 T7455 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 672268 T7455 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 672273 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 672273 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 672274 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 672274 T7455 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 672275 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 672276 T7449 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32752",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32752_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 672402 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 672507 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 672507 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 672507 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 673248 T7426 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47517_
   [junit4]   2> 673253 T7426 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47517_
   [junit4]   2> 673326 T7448 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 673326 T7470 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 673326 T7457 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 673339 T7471 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 673340 T7471 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 673354 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 673354 T7471 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 673354 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 673355 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 673356 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 673357 T7449 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47517",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47517_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 673357 T7449 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 673358 T7449 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 673387 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 673494 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 673494 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 673494 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 674355 T7471 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 674355 T7471 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/collection1
   [junit4]   2> 674355 T7471 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 674356 T7471 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 674357 T7471 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 674358 T7471 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 674358 T7471 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/collection1/'
   [junit4]   2> 674360 T7471 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/collection1/lib/classes/' to classloader
   [junit4]   2> 674360 T7471 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/collection1/lib/README' to classloader
   [junit4]   2> 674429 T7471 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 674506 T7471 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 674607 T7471 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 674618 T7471 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 675247 T7471 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 675252 T7471 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 675254 T7471 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 675260 T7471 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 675285 T7471 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 675285 T7471 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1392436430817/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty1/
   [junit4]   2> 675285 T7471 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c4b8217
   [junit4]   2> 675291 T7471 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty1
   [junit4]   2> 675292 T7471 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty1/index/
   [junit4]   2> 675293 T7471 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 675294 T7471 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty1/index
   [junit4]   2> 675295 T7471 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=36.970703125, floorSegmentMB=0.841796875, forceMergeDeletesPctAllowed=16.76668501874187, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21803891689922628
   [junit4]   2> 675313 T7471 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 675313 T7471 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 675320 T7471 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 675320 T7471 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 675321 T7471 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 675321 T7471 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 675321 T7471 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 675322 T7471 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 675322 T7471 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 675323 T7471 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 675323 T7471 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 675324 T7471 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 675325 T7471 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 675325 T7471 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 675325 T7471 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 675326 T7471 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 675327 T7471 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 675328 T7471 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 675328 T7471 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 675345 T7471 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 675350 T7471 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 675350 T7471 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 675351 T7471 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 675353 T7471 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 675353 T7471 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 675354 T7471 oass.SolrIndexSearcher.<init> Opening Searcher@1f514b44[collection1] main
   [junit4]   2> 675359 T7472 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f514b44[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 675364 T7471 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 675364 T7426 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
   [junit4]   2> 675365 T7426 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 675364 T7475 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47517 collection:collection1 shard:shard1
   [junit4]   2> 675367 T7475 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 675412 T7475 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 675432 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675433 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675433 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675433 T7475 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 675433 T7475 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C474 name=collection1 org.apache.solr.core.SolrCore@59f8ca10 url=http://127.0.0.1:47517/collection1 node=127.0.0.1:47517_ C474_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:47517, core=collection1, node_name=127.0.0.1:47517_}
   [junit4]   2> 675434 T7475 C474 P47517 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47517/collection1/
   [junit4]   2> 675434 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 675434 T7475 C474 P47517 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 675435 T7475 C474 P47517 oasc.SyncStrategy.syncToMe http://127.0.0.1:47517/collection1/ has no replicas
   [junit4]   2> 675435 T7475 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47517/collection1/ shard1
   [junit4]   2> 675435 T7475 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 675480 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675482 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675482 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675482 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675527 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 675540 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675643 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675643 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675643 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675679 T7475 oasc.ZkController.register We are http://127.0.0.1:47517/collection1/ and leader is http://127.0.0.1:47517/collection1/
   [junit4]   2> 675679 T7475 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47517
   [junit4]   2> 675679 T7475 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 675679 T7475 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 675680 T7475 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 675683 T7475 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 675684 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675684 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675685 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 675685 T7449 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47517",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47517_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 675694 T7426 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 675695 T7426 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 675698 T7426 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16379
   [junit4]   2> 675698 T7426 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 675699 T7426 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 675699 T7426 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662
   [junit4]   2> 675700 T7426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/'
   [junit4]   2> 675717 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 675740 T7426 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/solr.xml
   [junit4]   2> 675821 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675821 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675821 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 675839 T7426 oasc.CoreContainer.<init> New CoreContainer 866840363
   [junit4]   2> 675840 T7426 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/]
   [junit4]   2> 675841 T7426 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 675842 T7426 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 675842 T7426 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 675843 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 675843 T7426 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 675844 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 675844 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 675845 T7426 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 675845 T7426 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 675857 T7426 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 675858 T7426 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 675858 T7426 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 675859 T7426 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28338/solr
   [junit4]   2> 675859 T7426 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 675860 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 675866 T7486 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d8bcf3 name:ZooKeeperConnection Watcher:127.0.0.1:28338 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 675867 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 675870 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 675888 T7488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21d9f142 name:ZooKeeperConnection Watcher:127.0.0.1:28338/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 675888 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 675904 T7426 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 676909 T7426 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16379_
   [junit4]   2> 676916 T7426 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16379_
   [junit4]   2> 676921 T7457 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 676922 T7488 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 676921 T7470 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 676921 T7448 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 676959 T7489 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 676960 T7489 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 676962 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 676962 T7489 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 676962 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 676963 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 676963 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 676964 T7449 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:16379",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:16379_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 676965 T7449 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 676965 T7449 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 676993 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 677097 T7488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 677097 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 677097 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 677097 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 677963 T7489 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 677963 T7489 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/collection1
   [junit4]   2> 677964 T7489 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 677964 T7489 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 677965 T7489 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 677966 T7489 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 677966 T7489 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/collection1/'
   [junit4]   2> 677974 T7489 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/collection1/lib/classes/' to classloader
   [junit4]   2> 677975 T7489 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/collection1/lib/README' to classloader
   [junit4]   2> 678051 T7489 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 678129 T7489 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 678230 T7489 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 678242 T7489 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 678891 T7489 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 678895 T7489 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 678898 T7489 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 678904 T7489 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 678929 T7489 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 678929 T7489 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1392436434662/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty2/
   [junit4]   2> 678930 T7489 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6c4b8217
   [junit4]   2> 678936 T7489 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty2
   [junit4]   2> 678937 T7489 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty2/index/
   [junit4]   2> 678938 T7489 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 678939 T7489 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty2/index
   [junit4]   2> 678940 T7489 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=32, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=36.970703125, floorSegmentMB=0.841796875, forceMergeDeletesPctAllowed=16.76668501874187, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.21803891689922628
   [junit4]   2> 678948 T7489 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 678948 T7489 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 678955 T7489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 678955 T7489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 678956 T7489 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 678956 T7489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 678956 T7489 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 678957 T7489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 678957 T7489 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 678958 T7489 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 678958 T7489 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 678959 T7489 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 678960 T7489 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 678960 T7489 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 678961 T7489 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 678961 T7489 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 678962 T7489 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 678963 T7489 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 678964 T7489 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 678981 T7489 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 678985 T7489 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 678985 T7489 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 678987 T7489 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 678988 T7489 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392436426890/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 678989 T7489 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 678989 T7489 oass.SolrIndexSearcher.<init> Opening Searcher@3a38935c[collection1] main
   [junit4]   2> 678995 T7490 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a38935c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 678999 T7489 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 679000 T7426 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1
   [junit4]   2> 679000 T7426 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 679000 T7493 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:16379 collection:collection1 shard:shard2
   [junit4]   2> 679002 T7493 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 679037 T7493 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 679059 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679059 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679060 T7493 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 679060 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679060 T7493 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C475 name=collection1 org.apache.solr.core.SolrCore@2e9d2141 url=http://127.0.0.1:16379/collection1 node=127.0.0.1:16379_ C475_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:16379, core=collection1, node_name=127.0.0.1:16379_}
   [junit4]   2> 679060 T7493 C475 P16379 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:16379/collection1/
   [junit4]   2> 679061 T7493 C475 P16379 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 679061 T7493 C475 P16379 oasc.SyncStrategy.syncToMe http://127.0.0.1:16379/collection1/ has no replicas
   [junit4]   2> 679061 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 679061 T7493 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:16379/collection1/ shard2
   [junit4]   2> 679062 T7493 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 679075 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679081 T7488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679081 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679081 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679081 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679125 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 679149 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679253 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679253 T7488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679253 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679253 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679277 T7493 oasc.ZkController.register We are http://127.0.0.1:16379/collection1/ and leader is http://127.0.0.1:16379/collection1/
   [junit4]   2> 679277 T7493 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16379
   [junit4]   2> 679277 T7493 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 679277 T7493 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 679278 T7493 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 679279 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679279 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679280 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679280 T7493 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 679281 T7449 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 679282 T7449 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:16379",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:16379_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 679298 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 679328 T7426 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 679329 T7426 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 679332 T7426 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:56001
   [junit4]   2> 679333 T7426 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 679333 T7426 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 679334 T7426 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392436438297
   [junit4]   2> 679334 T7426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392436438297/'
   [junit4]   2> 679375 T7426 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392436438297/solr.xml
   [junit4]   2> 679403 T7448 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679403 T7488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679403 T7457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679403 T7470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 679476 T7426 oasc.CoreContainer.<init> New CoreContainer 1909042258
   [junit4]   2> 679477 T7426 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1392436438297/]
   [junit4]   2> 679479 T7426 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 679480 T7426 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 679480 T7426 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 679480 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 679481 T7426 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 679481 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 679482 T7426 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 679482 T7426 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 679483 T7426 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 679495 T7426 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 679495 T7426 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 679496 T7426 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 679497 T7426 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28338/solr
   [junit4]   2> 679497 T7426 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 679498 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 679501 T7504 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f5075da name:ZooKeeperConnection Watcher:127.0.0.1:28338 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 679501 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 679504 T7426 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 679518 T7506 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@632bdd15 name:ZooKeeperConnection Watcher:127.0.0.1:28338/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 679519 T7426 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 679524 T7426 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 680527 T7426 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56001_
   [junit4]   2> 680531 T7426 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56001_
   [junit4]   2> 680535 T7470 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 680535 T7488 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 680535 T7506 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 680535 T7457 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 680535 T7448 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 680557 T7507 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 680558 T7507 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 680559 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 680559 T7507 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 680560 T7448 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWa

[...truncated too long message...]

4]   1>       /solr/collections/collection1/leader_elect/shard1/election (0)
   [junit4]   1>     /solr/collections/collection1/shards (0)
   [junit4]   1>    /solr/collections/onenodecollection (2)
   [junit4]   1>    DATA:
   [junit4]   1>        {"configName":"conf1"}
   [junit4]   1>     /solr/collections/onenodecollection/leaders (0)
   [junit4]   1>     /solr/collections/onenodecollection/leader_elect (1)
   [junit4]   1>      /solr/collections/onenodecollection/leader_elect/shard1 (1)
   [junit4]   1>       /solr/collections/onenodecollection/leader_elect/shard1/election (0)
   [junit4]   1>   /solr/live_nodes (4)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:22193_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:34860_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:28110_ (0)
   [junit4]   1>    /solr/live_nodes/127.0.0.1:51263_ (0)
   [junit4]   1>   /solr/configs (1)
   [junit4]   1>    /solr/configs/conf1 (11)
   [junit4]   1>     /solr/configs/conf1/solrconfig.snippet.randomindexconfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/currency.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/schema.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/open-exchange-rates.json (0)
   [junit4]   1>     DATA:
   [junit4]   1>         {
   [junit4]   1>           "disclaimer": "This data is not real, it was synthetically created to match currency.xml.  It is modeled after the data format available from openexchangerates.org.  See https://openexchangerates.org/documentation for details",
   [junit4]   1>           "license": "http://www.apache.org/licenses/LICENSE-2.0",
   [junit4]   1>           "timestamp": 1332070464,
   [junit4]   1>         
   [junit4]   1>         
   [junit4]   1>           "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
   [junit4]   1>         
   [junit4]   1>         
   [junit4]   1>           "base": "USD",
   [junit4]   1>           "rates": {
   [junit4]   1>             "USD": 1,
   [junit4]   1>             "JPY": 81.29,
   [junit4]   1>             "EUR": 2.5,
   [junit4]   1>             "GBP": 0.5,
   [junit4]   1>             "MXN": 2.0
   [junit4]   1>           }
   [junit4]   1>         }
   [junit4]   1>         
   [junit4]   1>     /solr/configs/conf1/protwords.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/stopwords.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/enumsConfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/old_synonyms.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/solrconfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/synonyms.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>   /solr/overseer (3)
   [junit4]   1>   DATA:
   [junit4]   1>       
   [junit4]   1>    /solr/overseer/collection-queue-work (0)
   [junit4]   1>    /solr/overseer/queue-work (0)
   [junit4]   1>    /solr/overseer/queue (0)
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=8850D5789F557AF7 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en -Dtests.timezone=Africa/Casablanca -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    112s J1 | BasicDistributedZk2Test.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:22193/collection1, https://127.0.0.1:28110/collection1]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8850D5789F557AF7:9B65B60E80A1ACB]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:362)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1356)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:563)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:545)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:524)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:290)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:107)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: no servers hosting shard: 
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:293)
   [junit4]    > 	... 50 more
   [junit4]   2> 2035630 T42678 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 111744 T42677 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=en, timezone=Africa/Casablanca
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=283125776,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [DefaultValueUpdateProcessorTest, TestQuerySenderListener, TestMultiCoreConfBootstrap, StressHdfsTest, SolrCmdDistributorTest, DirectSolrConnectionTest, DateFieldTest, TestSchemaSimilarityResource, TestBinaryResponseWriter, DeleteReplicaTest, XmlUpdateRequestHandlerTest, NumericFieldsTest, ZkSolrClientTest, TestNoOpRegenerator, NoCacheHeaderTest, TestIBSimilarityFactory, UniqFieldsUpdateProcessorFactoryTest, TestArbitraryIndexDir, TestCursorMarkWithoutUniqueKey, TestSweetSpotSimilarityFactory, TestWordDelimiterFilterFactory, RequestHandlersTest, TestCSVResponseWriter, TestSimpleQParserPlugin, ShardSplitTest, TestStandardQParsers, ConvertedLegacyTest, TestSolr4Spatial, TermVectorComponentDistributedTest, BlockDirectoryTest, ConnectionManagerTest, TestDistributedGrouping, BadIndexSchemaTest, ZkNodePropsTest, TestSolrXml, TestMaxScoreQueryParser, SpellingQueryConverterTest, TestZkChroot, TestModifyConfFiles, EnumFieldTest, ShardRoutingCustomTest, AlternateDirectoryTest, TestDynamicFieldCollectionResource, OverseerRolesTest, SortByFunctionTest, TestDefaultSimilarityFactory, TestDistributedSearch, SuggesterWFSTTest, PrimUtilsTest, ChaosMonkeyNothingIsSafeTest, PingRequestHandlerTest, TestSolrQueryParserDefaultOperatorResource, MultiTermTest, TestUtils, HdfsCollectionsAPIDistributedZkTest, DocumentAnalysisRequestHandlerTest, HdfsUnloadDistributedZkTest, TestSolrXmlPersistence, SolrTestCaseJ4Test, TestXIncludeConfig, CachingDirectoryFactoryTest, PreAnalyzedUpdateProcessorTest, OpenCloseCoreStressTest, CurrencyFieldOpenExchangeTest, TestFuzzyAnalyzedSuggestions, FullSolrCloudDistribCmdsTest, TestDFRSimilarityFactory, DistribCursorPagingTest, TestCodecSupport, RequiredFieldsTest, HighlighterConfigTest, StatsComponentTest, TestRangeQuery, TimeZoneUtilsTest, OutputWriterTest, AssignTest, TestLazyCores, TestSchemaResource, NotRequiredUniqueKeyTest, TestCollationField, TestShardHandlerFactory, SolrCoreCheckLockOnStartupTest, PathHierarchyTokenizerFactoryTest, TestCoreDiscovery, DirectSolrSpellCheckerTest, TestDistribDocBasedVersion, FieldAnalysisRequestHandlerTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test]
   [junit4] Completed on J1 in 112.03s, 1 test, 1 error <<< FAILURES!

[...truncated 665 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:453: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:433: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1268: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:901: There were test failures: 372 suites, 1604 tests, 1 error, 1 failure, 35 ignored (7 assumptions)

Total time: 78 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message