lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2202 - Failure
Date Tue, 03 Dec 2013 05:36:26 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2202/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
There were expected update fails expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: There were expected update fails expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([F11DBB7B76C5D8E0:70FB3563019AB8DC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:202)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:843)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 10382 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 1974830 T5208 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1974836 T5208 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1386048502975
   [junit4]   2> 1974837 T5208 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1974838 T5209 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1974939 T5208 oasc.ZkTestServer.run start zk server on port:10787
   [junit4]   2> 1974940 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1974950 T5215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20f764f name:ZooKeeperConnection Watcher:127.0.0.1:10787 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1974950 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1974951 T5208 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1974956 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1974972 T5217 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@692f8eb5 name:ZooKeeperConnection Watcher:127.0.0.1:10787/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1974973 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1974973 T5208 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1974977 T5208 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1974979 T5208 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1974982 T5208 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1974984 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1974985 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1974990 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1974991 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1975098 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1975099 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1975102 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1975103 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1975110 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1975111 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1975114 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1975114 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1975118 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1975118 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1975121 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1975122 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1975130 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1975130 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1975151 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1975152 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1975155 T5208 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1975156 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1975449 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1975455 T5208 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:57910
   [junit4]   2> 1975456 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1975456 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1975457 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306
   [junit4]   2> 1975457 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/'
   [junit4]   2> 1975495 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/solr.xml
   [junit4]   2> 1975570 T5208 oasc.CoreContainer.<init> New CoreContainer 1624318877
   [junit4]   2> 1975571 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/]
   [junit4]   2> 1975573 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1975573 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1975574 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1975574 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1975575 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1975575 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1975576 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1975577 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1975577 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1975595 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1975595 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1975596 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1975596 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10787/solr
   [junit4]   2> 1975597 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1975598 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1975600 T5228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42794a13 name:ZooKeeperConnection Watcher:127.0.0.1:10787 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1975601 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1975604 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1975615 T5230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@988a88c name:ZooKeeperConnection Watcher:127.0.0.1:10787/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1975615 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1975617 T5208 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1975628 T5208 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1975631 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1975633 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57910_
   [junit4]   2> 1975635 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57910_
   [junit4]   2> 1975645 T5208 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1975661 T5208 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1975664 T5208 oasc.Overseer.start Overseer (id=90836074691493891-127.0.0.1:57910_-n_0000000000) starting
   [junit4]   2> 1975676 T5208 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1975687 T5232 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1975688 T5208 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1975690 T5208 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1975692 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1975694 T5231 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1975699 T5233 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1975700 T5233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1975701 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1975701 T5233 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1975702 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1975703 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57910",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57910_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1975703 T5231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1975703 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1975712 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1975713 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1976702 T5233 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1976702 T5233 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/collection1
   [junit4]   2> 1976702 T5233 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1976703 T5233 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1976703 T5233 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1976704 T5233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/collection1/'
   [junit4]   2> 1976706 T5233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/collection1/lib/README' to classloader
   [junit4]   2> 1976706 T5233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/collection1/lib/classes/' to classloader
   [junit4]   2> 1976769 T5233 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1976829 T5233 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1976931 T5233 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1976941 T5233 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1977510 T5233 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1977514 T5233 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1977517 T5233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1977522 T5233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1977542 T5233 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1977542 T5233 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386048503306/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/control/data/
   [junit4]   2> 1977543 T5233 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c4ef185
   [junit4]   2> 1977544 T5233 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/control/data
   [junit4]   2> 1977545 T5233 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/control/data/index/
   [junit4]   2> 1977546 T5233 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1977547 T5233 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/control/data/index
   [junit4]   2> 1977548 T5233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=27.16796875, floorSegmentMB=0.34375, forceMergeDeletesPctAllowed=13.940182289736686, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1977553 T5233 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1977553 T5233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1977556 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1977556 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1977557 T5233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1977557 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1977557 T5233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1977557 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1977558 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1977558 T5233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1977559 T5233 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1977559 T5233 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1977560 T5233 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1977560 T5233 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1977560 T5233 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1977561 T5233 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1977561 T5233 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1977562 T5233 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1977571 T5233 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1977574 T5233 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1977575 T5233 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1977575 T5233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=23.421875, floorSegmentMB=1.71484375, forceMergeDeletesPctAllowed=17.922051447078363, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1977577 T5233 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1977577 T5233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1977578 T5233 oass.SolrIndexSearcher.<init> Opening Searcher@3af7970 main
   [junit4]   2> 1977580 T5234 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3af7970 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1977582 T5233 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1977583 T5233 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57910 collection:control_collection shard:shard1
   [junit4]   2> 1977583 T5233 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1977604 T5233 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1977613 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1977613 T5233 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1977613 T5233 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1451 name=collection1 org.apache.solr.core.SolrCore@f84e48a url=http://127.0.0.1:57910/collection1 node=127.0.0.1:57910_ C1451_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:57910, core=collection1, node_name=127.0.0.1:57910_}
   [junit4]   2> 1977614 T5233 C1451 P57910 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57910/collection1/
   [junit4]   2> 1977614 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1977614 T5233 C1451 P57910 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1977614 T5233 C1451 P57910 oasc.SyncStrategy.syncToMe http://127.0.0.1:57910/collection1/ has no replicas
   [junit4]   2> 1977615 T5233 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57910/collection1/ shard1
   [junit4]   2> 1977615 T5233 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1977623 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1977631 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1977645 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1977652 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1977755 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1977797 T5233 oasc.ZkController.register We are http://127.0.0.1:57910/collection1/ and leader is http://127.0.0.1:57910/collection1/
   [junit4]   2> 1977797 T5233 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57910
   [junit4]   2> 1977797 T5233 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1977797 T5233 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1977798 T5233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1977799 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1977799 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1977799 T5233 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1977800 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1977801 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1977801 T5208 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
   [junit4]   2> 1977802 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1977801 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57910",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57910_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1977804 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1977805 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1977814 T5237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18519df name:ZooKeeperConnection Watcher:127.0.0.1:10787/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1977814 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1977815 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1977817 T5208 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1977907 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1977907 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1978074 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1978077 T5208 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:51481
   [junit4]   2> 1978077 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1978078 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1978078 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956
   [junit4]   2> 1978079 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/'
   [junit4]   2> 1978114 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/solr.xml
   [junit4]   2> 1978183 T5208 oasc.CoreContainer.<init> New CoreContainer 926893340
   [junit4]   2> 1978184 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/]
   [junit4]   2> 1978186 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1978186 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1978187 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1978187 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1978188 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1978188 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1978189 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1978189 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1978189 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1978206 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1978206 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1978207 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1978207 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10787/solr
   [junit4]   2> 1978208 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1978209 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1978211 T5248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ad24d9a name:ZooKeeperConnection Watcher:127.0.0.1:10787 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1978212 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1978223 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1978225 T5250 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18afd698 name:ZooKeeperConnection Watcher:127.0.0.1:10787/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1978225 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1978234 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1979237 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51481_
   [junit4]   2> 1979248 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51481_
   [junit4]   2> 1979251 T5237 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1979251 T5230 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1979251 T5250 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1979259 T5251 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1979259 T5251 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1979260 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1979260 T5251 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1979261 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1979261 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1979262 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1979263 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51481",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51481_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1979263 T5231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1979264 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1979266 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1979375 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1979375 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1979375 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1980262 T5251 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1980262 T5251 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/collection1
   [junit4]   2> 1980262 T5251 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1980263 T5251 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1980264 T5251 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1980264 T5251 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/collection1/'
   [junit4]   2> 1980266 T5251 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/collection1/lib/README' to classloader
   [junit4]   2> 1980267 T5251 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/collection1/lib/classes/' to classloader
   [junit4]   2> 1980329 T5251 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1980389 T5251 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1980491 T5251 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1980501 T5251 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1981065 T5251 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1981070 T5251 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1981072 T5251 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1981077 T5251 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1981097 T5251 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1981097 T5251 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386048505956/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty1/
   [junit4]   2> 1981097 T5251 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c4ef185
   [junit4]   2> 1981099 T5251 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty1
   [junit4]   2> 1981100 T5251 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty1/index/
   [junit4]   2> 1981100 T5251 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1981103 T5251 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty1/index
   [junit4]   2> 1981103 T5251 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=27.16796875, floorSegmentMB=0.34375, forceMergeDeletesPctAllowed=13.940182289736686, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1981108 T5251 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1981108 T5251 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1981112 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1981112 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1981112 T5251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1981113 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1981113 T5251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1981113 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1981114 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1981114 T5251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1981115 T5251 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1981115 T5251 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1981116 T5251 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1981116 T5251 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1981117 T5251 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1981118 T5251 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1981118 T5251 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1981119 T5251 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1981128 T5251 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1981132 T5251 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1981133 T5251 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1981134 T5251 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=23.421875, floorSegmentMB=1.71484375, forceMergeDeletesPctAllowed=17.922051447078363, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1981135 T5251 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1981136 T5251 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1981136 T5251 oass.SolrIndexSearcher.<init> Opening Searcher@7ddc7dc3 main
   [junit4]   2> 1981140 T5252 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ddc7dc3 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1981143 T5251 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1981143 T5251 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51481 collection:collection1 shard:shard1
   [junit4]   2> 1981144 T5251 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1981160 T5251 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1981163 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981163 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981163 T5251 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1981163 T5251 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1981163 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C1452 name=collection1 org.apache.solr.core.SolrCore@25074393 url=http://127.0.0.1:51481/collection1 node=127.0.0.1:51481_ C1452_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51481, core=collection1, node_name=127.0.0.1:51481_}
   [junit4]   2> 1981164 T5251 C1452 P51481 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51481/collection1/
   [junit4]   2> 1981164 T5251 C1452 P51481 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1981165 T5251 C1452 P51481 oasc.SyncStrategy.syncToMe http://127.0.0.1:51481/collection1/ has no replicas
   [junit4]   2> 1981165 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1981165 T5251 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51481/collection1/ shard1
   [junit4]   2> 1981166 T5251 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1981173 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981180 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981180 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981180 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981196 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1981200 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981303 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981303 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981303 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981352 T5251 oasc.ZkController.register We are http://127.0.0.1:51481/collection1/ and leader is http://127.0.0.1:51481/collection1/
   [junit4]   2> 1981352 T5251 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51481
   [junit4]   2> 1981352 T5251 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1981352 T5251 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1981353 T5251 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1981359 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981359 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981359 T5251 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1981359 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981360 T5208 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
   [junit4]   2> 1981361 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1981361 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1981362 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51481",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51481_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1981365 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1981467 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981467 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981467 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1981638 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1981641 T5208 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:44963
   [junit4]   2> 1981641 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1981642 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1981643 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500
   [junit4]   2> 1981643 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/'
   [junit4]   2> 1981689 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/solr.xml
   [junit4]   2> 1981764 T5208 oasc.CoreContainer.<init> New CoreContainer 886125905
   [junit4]   2> 1981765 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/]
   [junit4]   2> 1981767 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1981767 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1981768 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1981768 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1981769 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1981769 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1981770 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1981770 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1981771 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1981788 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1981789 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1981789 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1981790 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10787/solr
   [junit4]   2> 1981791 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1981792 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1981794 T5264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@130373ee name:ZooKeeperConnection Watcher:127.0.0.1:10787 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1981795 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1981798 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1981805 T5266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28047ac1 name:ZooKeeperConnection Watcher:127.0.0.1:10787/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1981806 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1981810 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1982813 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44963_
   [junit4]   2> 1982815 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44963_
   [junit4]   2> 1982818 T5237 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1982818 T5230 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1982818 T5266 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1982818 T5250 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1982836 T5267 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1982836 T5267 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1982838 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1982838 T5267 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1982838 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1982838 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1982840 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1982840 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44963",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44963_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1982841 T5231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1982841 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1982844 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1982946 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1982946 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1982946 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1982946 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1983839 T5267 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1983839 T5267 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/collection1
   [junit4]   2> 1983839 T5267 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1983840 T5267 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1983840 T5267 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1983841 T5267 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/collection1/'
   [junit4]   2> 1983843 T5267 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/collection1/lib/classes/' to classloader
   [junit4]   2> 1983844 T5267 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/collection1/lib/README' to classloader
   [junit4]   2> 1983906 T5267 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1983967 T5267 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1984069 T5267 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1984079 T5267 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1984648 T5267 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1984653 T5267 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1984655 T5267 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1984660 T5267 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1984680 T5267 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1984680 T5267 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386048509500/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty2/
   [junit4]   2> 1984681 T5267 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c4ef185
   [junit4]   2> 1984682 T5267 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty2
   [junit4]   2> 1984683 T5267 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty2/index/
   [junit4]   2> 1984684 T5267 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1984686 T5267 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty2/index
   [junit4]   2> 1984687 T5267 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=27.16796875, floorSegmentMB=0.34375, forceMergeDeletesPctAllowed=13.940182289736686, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1984691 T5267 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1984691 T5267 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1984695 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1984695 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1984696 T5267 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1984696 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1984696 T5267 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1984697 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1984697 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1984698 T5267 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1984698 T5267 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1984699 T5267 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1984699 T5267 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1984700 T5267 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1984700 T5267 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1984701 T5267 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1984701 T5267 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1984702 T5267 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1984712 T5267 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1984716 T5267 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1984716 T5267 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1984717 T5267 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=23.421875, floorSegmentMB=1.71484375, forceMergeDeletesPctAllowed=17.922051447078363, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1984718 T5267 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1984719 T5267 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1984719 T5267 oass.SolrIndexSearcher.<init> Opening Searcher@63fe6bd8 main
   [junit4]   2> 1984723 T5268 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63fe6bd8 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1984726 T5267 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1984726 T5267 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44963 collection:collection1 shard:shard2
   [junit4]   2> 1984727 T5267 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1984745 T5267 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1984748 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1984748 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1984748 T5267 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1984748 T5267 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1984748 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C1453 name=collection1 org.apache.solr.core.SolrCore@268728bd url=http://127.0.0.1:44963/collection1 node=127.0.0.1:44963_ C1453_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:44963, core=collection1, node_name=127.0.0.1:44963_}
   [junit4]   2> 1984749 T5267 C1453 P44963 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44963/collection1/
   [junit4]   2> 1984749 T5267 C1453 P44963 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1984750 T5267 C1453 P44963 oasc.SyncStrategy.syncToMe http://127.0.0.1:44963/collection1/ has no replicas
   [junit4]   2> 1984750 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1984750 T5267 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44963/collection1/ shard2
   [junit4]   2> 1984751 T5267 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1984753 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1984754 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984754 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984754 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984754 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984757 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1984760 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1984862 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984862 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984862 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984862 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1984910 T5267 oasc.ZkController.register We are http://127.0.0.1:44963/collection1/ and leader is http://127.0.0.1:44963/collection1/
   [junit4]   2> 1984910 T5267 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44963
   [junit4]   2> 1984910 T5267 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1984910 T5267 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1984911 T5267 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1984912 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1984912 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1984912 T5267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1984912 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1984914 T5208 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
   [junit4]   2> 1984914 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1984914 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1984915 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44963",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44963_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1984918 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1985024 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1985024 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1985024 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1985024 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1985214 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1985217 T5208 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:57699
   [junit4]   2> 1985217 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1985218 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1985218 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054
   [junit4]   2> 1985219 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/'
   [junit4]   2> 1985257 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/solr.xml
   [junit4]   2> 1985340 T5208 oasc.CoreContainer.<init> New CoreContainer 192304346
   [junit4]   2> 1985341 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/]
   [junit4]   2> 1985343 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1985343 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1985344 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1985344 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1985345 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1985345 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1985346 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1985346 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1985347 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1985364 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1985365 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1985365 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1985366 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10787/solr
   [junit4]   2> 1985367 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1985368 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1985374 T5280 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bcba17 name:ZooKeeperConnection Watcher:127.0.0.1:10787 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1985375 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1985378 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1985385 T5282 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19c523ec name:ZooKeeperConnection Watcher:127.0.0.1:10787/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1985386 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1985390 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1986393 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57699_
   [junit4]   2> 1986395 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57699_
   [junit4]   2> 1986398 T5237 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1986398 T5266 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1986398 T5250 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1986398 T5282 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1986398 T5230 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1986414 T5283 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1986414 T5283 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1986415 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1986415 T5283 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1986416 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1986416 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1986417 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1986418 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57699",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57699_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1986418 T5231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1986418 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1986421 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1986524 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1986524 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1986524 T5282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1986524 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1986524 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1987417 T5283 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1987417 T5283 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/collection1
   [junit4]   2> 1987417 T5283 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1987418 T5283 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1987419 T5283 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1987420 T5283 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/collection1/'
   [junit4]   2> 1987421 T5283 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/collection1/lib/classes/' to classloader
   [junit4]   2> 1987422 T5283 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/collection1/lib/README' to classloader
   [junit4]   2> 1987486 T5283 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1987548 T5283 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1987650 T5283 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1987660 T5283 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1988226 T5283 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1988231 T5283 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1988233 T5283 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1988238 T5283 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1988258 T5283 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1988258 T5283 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386048513054/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty3/
   [junit4]   2> 1988259 T5283 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5c4ef185
   [junit4]   2> 1988260 T5283 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty3
   [junit4]   2> 1988261 T5283 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386048502974/jetty3/index/
   [junit4]   2> 1988262 T5283 oasc.SolrCore.i

[...truncated too long message...]

LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 2154374 T5353 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/90836074691493906-core_node7-n_0000000003
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:169)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:166)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:166)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:66)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:280)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2154374 T5353 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2154375 T5353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2154375 T5353 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2154375 T5353 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2154421 T5335 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=de_CH, timezone=Asia/Nicosia
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=302372104,total=476381184
   [junit4]   2> NOTE: All tests run in this JVM: [CoreMergeIndexesAdminHandlerTest, TestDynamicFieldResource, TestXIncludeConfig, TestDistributedSearch, LeaderElectionIntegrationTest, TestRecoveryHdfs, TestNRTOpen, ZkSolrClientTest, SpellCheckCollatorTest, TestFoldingMultitermQuery, SolrTestCaseJ4Test, PingRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestFastWriter, TestStressLucene, BadIndexSchemaTest, NotRequiredUniqueKeyTest, OpenExchangeRatesOrgProviderTest, OutputWriterTest, PluginInfoTest, HardAutoCommitTest, TestJmxIntegration, TestStressRecovery, TestIndexingPerformance, SpellCheckComponentTest, SolrPluginUtilsTest, CurrencyFieldOpenExchangeTest, TestSweetSpotSimilarityFactory, TestFuzzyAnalyzedSuggestions, TestStressReorder, TestSolrQueryParserDefaultOperatorResource, SuggesterTSTTest, HighlighterTest, TestSolrDeletionPolicy2, DefaultValueUpdateProcessorTest, TestCollationKeyRangeQueries, TestRecovery, BasicZkTest, TermVectorComponentDistributedTest, TestZkChroot, TestFieldTypeResource, ChaosMonkeySafeLeaderTest, IndexSchemaRuntimeFieldTest, UniqFieldsUpdateProcessorFactoryTest, ZkNodePropsTest, IndexBasedSpellCheckerTest, TestSimpleQParserPlugin, XmlUpdateRequestHandlerTest, HdfsUnloadDistributedZkTest, NoCacheHeaderTest, StandardRequestHandlerTest, SpellingQueryConverterTest, DocumentAnalysisRequestHandlerTest, TestQuerySenderListener, DirectUpdateHandlerTest, TestBinaryField, TestRandomFaceting, TestPartialUpdateDeduplication, TestImplicitCoreProperties, TestPhraseSuggestions, PrimUtilsTest, CollectionsAPIDistributedZkTest, DeleteInactiveReplicaTest, TestCollationField, SimplePostToolTest, TestArbitraryIndexDir, SuggestComponentTest, TestLazyCores, TestRTGBase, TestFieldResource, SolrCoreCheckLockOnStartupTest, HdfsCollectionsAPIDistributedZkTest, TestDFRSimilarityFactory, TestComponentsName, TestUtils, AliasIntegrationTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestReload, TestFastLRUCache, DocValuesMissingTest, RequiredFieldsTest, AddBlockUpdateTest, DistanceFunctionTest, TestCSVLoader, OpenCloseCoreStressTest, OverseerTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, LeaderElectionTest, ZkCLITest, ShardRoutingCustomTest, TestDistributedGrouping, TestFaceting, TestHashPartitioner, DistributedSpellCheckComponentTest, ZkControllerTest, TestRealTimeGet, TestMultiCoreConfBootstrap, DistributedTermsComponentTest, TestRangeQuery, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, StatsComponentTest, SolrCmdDistributorTest, ConvertedLegacyTest, TestFiltering, OverseerCollectionProcessorTest, TestIndexSearcher, SoftAutoCommitTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, CoreAdminHandlerTest, SolrRequestParserTest, SuggesterTest, SpatialFilterTest, SuggesterWFSTTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestWriterPerf, DirectSolrSpellCheckerTest, TestQueryTypes, PrimitiveFieldTypeTest, TestOmitPositions, FileBasedSpellCheckerTest, TermsComponentTest, RequestHandlersTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, IndexSchemaTest, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, TestCSVResponseWriter, TestAnalyzedSuggestions, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, SearchHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, UpdateParamsTest, AlternateDirectoryTest, TestQuerySenderNoQuery, ResponseLogComponentTest, SolrIndexConfigTest, MultiTermTest, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, ExternalFileFieldSortTest, TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion, TestCharFilters, TestCodecSupport, EchoParamsTest, TestPerFieldSimilarity, TestNumberUtils, TestIBSimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, TimeZoneUtilsTest, TestFastOutputStream, ScriptEngineTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J1 in 180.47s, 1 test, 1 failure <<< FAILURES!

[...truncated 159 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:426: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:406: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1279: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:912: There were test failures: 347 suites, 1515 tests, 1 failure, 34 ignored (6 assumptions)

Total time: 68 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message