lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2242 - Still Failing
Date Sun, 29 Dec 2013 00:25:31 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2242/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
There were expected update fails expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: There were expected update fails expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([262A32FED6183C06:A7CCBCE6A1475C3A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:202)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:849)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 10274 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 1665814 T3204 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1665820 T3204 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1388276348573
   [junit4]   2> 1665821 T3204 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1665822 T3205 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1665923 T3204 oasc.ZkTestServer.run start zk server on port:42233
   [junit4]   2> 1665925 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1665930 T3212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56261adf name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1665930 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1665931 T3204 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1665981 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1665983 T3215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66736dcc name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1665984 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1665984 T3204 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1665993 T3204 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1665996 T3204 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1665999 T3204 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1666007 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1666008 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1666021 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1666022 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1666127 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1666128 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1666137 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1666138 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1666142 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1666143 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1666146 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1666147 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1666150 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1666151 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1666155 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1666156 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1666160 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1666160 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1666164 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1666164 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1666168 T3204 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1666169 T3204 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1666467 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1666471 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28670
   [junit4]   2> 1666471 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1666472 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1666472 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925
   [junit4]   2> 1666473 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/'
   [junit4]   2> 1666513 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/solr.xml
   [junit4]   2> 1666594 T3204 oasc.CoreContainer.<init> New CoreContainer 1445083937
   [junit4]   2> 1666595 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/]
   [junit4]   2> 1666597 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1666598 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1666598 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1666599 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1666599 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1666599 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1666600 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1666600 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1666601 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1666617 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1666618 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1666619 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1666619 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
   [junit4]   2> 1666620 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1666621 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1666624 T3227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60899459 name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1666624 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1666627 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1666636 T3230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e6f0d5e name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1666636 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1666638 T3204 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1666649 T3204 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1666653 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1666655 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28670_
   [junit4]   2> 1666662 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28670_
   [junit4]   2> 1666666 T3204 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1666680 T3204 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1666683 T3204 oasc.Overseer.start Overseer (id=90982078780538883-127.0.0.1:28670_-n_0000000000) starting
   [junit4]   2> 1666694 T3204 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1666705 T3232 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1666706 T3204 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1666709 T3204 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1666711 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1666714 T3231 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1666720 T3233 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1666720 T3233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1666721 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1666721 T3233 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1666723 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1666724 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:28670",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:28670_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1666724 T3231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1666724 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1666727 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1666737 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1667723 T3233 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1667723 T3233 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1
   [junit4]   2> 1667724 T3233 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1667725 T3233 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1667725 T3233 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1667726 T3233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/'
   [junit4]   2> 1667728 T3233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/lib/classes/' to classloader
   [junit4]   2> 1667729 T3233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/lib/README' to classloader
   [junit4]   2> 1667796 T3233 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1667846 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
   [junit4]   2> 1667871 T3233 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1667972 T3233 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1667995 T3233 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1668541 T3233 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1668545 T3233 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1668547 T3233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1668553 T3233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1668578 T3233 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1668621 T3233 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1388276348925/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/
   [junit4]   2> 1668621 T3233 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e14093f
   [junit4]   2> 1668623 T3233 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data
   [junit4]   2> 1668624 T3233 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index/
   [junit4]   2> 1668625 T3233 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1668626 T3233 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index
   [junit4]   2> 1668627 T3233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=316730711, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1668657 T3233 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1668657 T3233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1668660 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1668661 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1668661 T3233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1668661 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1668662 T3233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1668662 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1668662 T3233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1668663 T3233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1668663 T3233 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1668663 T3233 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1668664 T3233 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1668664 T3233 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1668665 T3233 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1668665 T3233 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1668665 T3233 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1668666 T3233 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1668675 T3233 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1668679 T3233 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1668679 T3233 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1668680 T3233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=69.9013671875, floorSegmentMB=0.7578125, forceMergeDeletesPctAllowed=3.0481504002723367, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1668681 T3233 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1668682 T3233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1668682 T3233 oass.SolrIndexSearcher.<init> Opening Searcher@7185d704 main
   [junit4]   2> 1668685 T3234 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7185d704 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1668686 T3233 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1668687 T3204 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
   [junit4]   2> 1668688 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1668687 T3237 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28670 collection:control_collection shard:shard1
   [junit4]   2> 1668689 T3237 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1668691 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1668702 T3240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78bc865c name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1668702 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1668704 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1668719 T3237 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1668719 T3204 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1668722 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1668722 T3237 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1668722 T3237 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1216 name=collection1 org.apache.solr.core.SolrCore@4475f8a0 url=http://127.0.0.1:28670/collection1 node=127.0.0.1:28670_ C1216_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:28670, core=collection1, node_name=127.0.0.1:28670_}
   [junit4]   2> 1668723 T3237 C1216 P28670 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28670/collection1/
   [junit4]   2> 1668723 T3237 C1216 P28670 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1668723 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1668723 T3237 C1216 P28670 oasc.SyncStrategy.syncToMe http://127.0.0.1:28670/collection1/ has no replicas
   [junit4]   2> 1668724 T3237 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28670/collection1/ shard1
   [junit4]   2> 1668724 T3237 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1668728 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1668739 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1668744 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1668759 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1668764 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1668868 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1668868 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1668912 T3237 oasc.ZkController.register We are http://127.0.0.1:28670/collection1/ and leader is http://127.0.0.1:28670/collection1/
   [junit4]   2> 1668912 T3237 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28670
   [junit4]   2> 1668912 T3237 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1668912 T3237 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1668912 T3237 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1668914 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1668914 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1668914 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1668915 T3237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1668916 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1668917 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:28670",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:28670_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1668920 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1669004 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1669006 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:37265
   [junit4]   2> 1669007 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1669008 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1669008 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471
   [junit4]   2> 1669009 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/'
   [junit4]   2> 1669024 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1669024 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1669050 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/solr.xml
   [junit4]   2> 1669130 T3204 oasc.CoreContainer.<init> New CoreContainer 822232844
   [junit4]   2> 1669131 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/]
   [junit4]   2> 1669133 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1669133 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1669134 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1669134 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1669134 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1669135 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1669135 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1669136 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1669136 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1669154 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1669155 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1669155 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1669156 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
   [junit4]   2> 1669156 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1669158 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1669185 T3252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51b7e98d name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1669186 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1669190 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1669200 T3255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a26c38c name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1669200 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1669206 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1670209 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37265_
   [junit4]   2> 1670219 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37265_
   [junit4]   2> 1670224 T3240 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1670224 T3255 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1670224 T3230 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1670239 T3256 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1670240 T3256 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1670241 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1670241 T3256 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1670242 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1670242 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1670243 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1670244 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37265",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37265_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1670244 T3231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1670244 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1670248 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1670387 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1670388 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1670387 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1671243 T3256 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1671243 T3256 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1
   [junit4]   2> 1671243 T3256 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1671244 T3256 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1671245 T3256 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1671246 T3256 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/'
   [junit4]   2> 1671248 T3256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/lib/classes/' to classloader
   [junit4]   2> 1671249 T3256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/lib/README' to classloader
   [junit4]   2> 1671312 T3256 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1671376 T3256 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1671477 T3256 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1671499 T3256 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1672077 T3256 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1672081 T3256 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1672084 T3256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1672089 T3256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1672119 T3256 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1672119 T3256 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1388276351471/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/
   [junit4]   2> 1672120 T3256 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e14093f
   [junit4]   2> 1672121 T3256 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1
   [junit4]   2> 1672122 T3256 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index/
   [junit4]   2> 1672123 T3256 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1672124 T3256 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index
   [junit4]   2> 1672124 T3256 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=316730711, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1672165 T3256 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1672166 T3256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1672170 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1672171 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1672171 T3256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1672171 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1672172 T3256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1672172 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1672173 T3256 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1672173 T3256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1672173 T3256 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1672174 T3256 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1672175 T3256 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1672175 T3256 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1672175 T3256 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1672176 T3256 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1672176 T3256 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1672177 T3256 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1672187 T3256 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1672191 T3256 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1672192 T3256 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1672193 T3256 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=69.9013671875, floorSegmentMB=0.7578125, forceMergeDeletesPctAllowed=3.0481504002723367, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1672194 T3256 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1672195 T3256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1672195 T3256 oass.SolrIndexSearcher.<init> Opening Searcher@7843566e main
   [junit4]   2> 1672199 T3257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7843566e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1672201 T3256 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1672202 T3204 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
   [junit4]   2> 1672203 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1672202 T3260 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37265 collection:collection1 shard:shard1
   [junit4]   2> 1672204 T3260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1672245 T3260 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1672256 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672257 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672257 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672257 T3260 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1672257 T3260 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1217 name=collection1 org.apache.solr.core.SolrCore@6327c579 url=http://127.0.0.1:37265/collection1 node=127.0.0.1:37265_ C1217_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:37265, core=collection1, node_name=127.0.0.1:37265_}
   [junit4]   2> 1672258 T3260 C1217 P37265 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37265/collection1/
   [junit4]   2> 1672258 T3260 C1217 P37265 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1672258 T3260 C1217 P37265 oasc.SyncStrategy.syncToMe http://127.0.0.1:37265/collection1/ has no replicas
   [junit4]   2> 1672258 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1672259 T3260 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37265/collection1/ shard1
   [junit4]   2> 1672259 T3260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1672297 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672340 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672340 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672340 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672381 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1672385 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672491 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1672493 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672494 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:18099
   [junit4]   2> 1672493 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672495 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1672493 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672496 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1672496 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956
   [junit4]   2> 1672497 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/'
   [junit4]   2> 1672533 T3260 oasc.ZkController.register We are http://127.0.0.1:37265/collection1/ and leader is http://127.0.0.1:37265/collection1/
   [junit4]   2> 1672533 T3260 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37265
   [junit4]   2> 1672533 T3260 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1672533 T3260 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1672533 T3260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1672535 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/solr.xml
   [junit4]   2> 1672537 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672537 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672538 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672538 T3260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1672539 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1672540 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:37265",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:37265_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1672543 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1672613 T3204 oasc.CoreContainer.<init> New CoreContainer 1441207837
   [junit4]   2> 1672614 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/]
   [junit4]   2> 1672616 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1672616 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1672617 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1672617 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1672618 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1672618 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1672619 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1672619 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1672619 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1672637 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1672638 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1672639 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1672639 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
   [junit4]   2> 1672640 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1672641 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1672648 T3272 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48da33b6 name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1672649 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1672680 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672680 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672680 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1672709 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1672725 T3275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78dd552e name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1672725 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1672730 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1673734 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:18099_
   [junit4]   2> 1673763 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:18099_
   [junit4]   2> 1673766 T3240 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1673767 T3230 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1673767 T3275 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1673767 T3255 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1673783 T3276 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1673783 T3276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1673785 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1673785 T3276 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1673785 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1673785 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1673786 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1673787 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:18099",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:18099_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1673787 T3231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1673788 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1673791 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1673894 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1673894 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1673894 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1673894 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1674786 T3276 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1674786 T3276 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1
   [junit4]   2> 1674786 T3276 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1674787 T3276 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1674788 T3276 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1674789 T3276 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/'
   [junit4]   2> 1674790 T3276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/lib/README' to classloader
   [junit4]   2> 1674791 T3276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/lib/classes/' to classloader
   [junit4]   2> 1674856 T3276 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1674921 T3276 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1675022 T3276 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1675044 T3276 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1675678 T3276 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1675683 T3276 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1675686 T3276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1675692 T3276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1675717 T3276 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1675718 T3276 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1388276354956/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/
   [junit4]   2> 1675718 T3276 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e14093f
   [junit4]   2> 1675720 T3276 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2
   [junit4]   2> 1675721 T3276 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index/
   [junit4]   2> 1675722 T3276 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1675724 T3276 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index
   [junit4]   2> 1675725 T3276 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=316730711, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1675729 T3276 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1675730 T3276 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1675734 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1675734 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1675735 T3276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1675735 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1675735 T3276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1675736 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1675736 T3276 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1675737 T3276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1675737 T3276 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1675738 T3276 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1675738 T3276 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1675738 T3276 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1675739 T3276 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1675739 T3276 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1675740 T3276 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1675741 T3276 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1675754 T3276 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1675758 T3276 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1675759 T3276 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1675760 T3276 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=69.9013671875, floorSegmentMB=0.7578125, forceMergeDeletesPctAllowed=3.0481504002723367, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1675762 T3276 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1675762 T3276 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1675762 T3276 oass.SolrIndexSearcher.<init> Opening Searcher@3aebe998 main
   [junit4]   2> 1675767 T3277 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3aebe998 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1675769 T3276 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1675770 T3204 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
   [junit4]   2> 1675770 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1675770 T3280 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:18099 collection:collection1 shard:shard2
   [junit4]   2> 1675772 T3280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1675788 T3280 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1675796 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675796 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675796 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675796 T3280 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1675797 T3280 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1218 name=collection1 org.apache.solr.core.SolrCore@1b62e6e5 url=http://127.0.0.1:18099/collection1 node=127.0.0.1:18099_ C1218_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:18099, core=collection1, node_name=127.0.0.1:18099_}
   [junit4]   2> 1675797 T3280 C1218 P18099 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:18099/collection1/
   [junit4]   2> 1675797 T3280 C1218 P18099 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1675797 T3280 C1218 P18099 oasc.SyncStrategy.syncToMe http://127.0.0.1:18099/collection1/ has no replicas
   [junit4]   2> 1675798 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1675798 T3280 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:18099/collection1/ shard2
   [junit4]   2> 1675798 T3280 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1675802 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675811 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675811 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675811 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675811 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675823 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1675832 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675935 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675935 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675935 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675935 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1675974 T3280 oasc.ZkController.register We are http://127.0.0.1:18099/collection1/ and leader is http://127.0.0.1:18099/collection1/
   [junit4]   2> 1675974 T3280 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:18099
   [junit4]   2> 1675974 T3280 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1675974 T3280 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1675975 T3280 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1675976 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675977 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675977 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1675977 T3280 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1675979 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1675979 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:18099",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:18099_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1675983 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1676088 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1676089 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1676089 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1676088 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1676120 T3204 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1676123 T3204 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:47019
   [junit4]   2> 1676123 T3204 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1676124 T3204 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1676124 T3204 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524
   [junit4]   2> 1676125 T3204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/'
   [junit4]   2> 1676161 T3204 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/solr.xml
   [junit4]   2> 1676234 T3204 oasc.CoreContainer.<init> New CoreContainer 498415665
   [junit4]   2> 1676234 T3204 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/]
   [junit4]   2> 1676236 T3204 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1676237 T3204 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1676237 T3204 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1676237 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1676238 T3204 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1676238 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1676239 T3204 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1676239 T3204 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1676240 T3204 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1676255 T3204 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1676255 T3204 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1676256 T3204 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1676256 T3204 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42233/solr
   [junit4]   2> 1676257 T3204 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1676258 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1676260 T3292 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13335fcf name:ZooKeeperConnection Watcher:127.0.0.1:42233 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1676261 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1676265 T3204 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1676272 T3295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a6c2a6 name:ZooKeeperConnection Watcher:127.0.0.1:42233/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1676272 T3204 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1676277 T3204 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1677281 T3204 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47019_
   [junit4]   2> 1677284 T3204 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47019_
   [junit4]   2> 1677288 T3240 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1677288 T3230 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1677288 T3275 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1677288 T3255 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1677289 T3295 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1677298 T3296 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1677298 T3296 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1677300 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677300 T3296 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1677300 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677301 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677301 T3231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1677302 T3231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47019",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47019_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1677303 T3231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1677303 T3231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1677306 T3230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1677408 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1677409 T3275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1677409 T3295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1677409 T3255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1677408 T3230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1678301 T3296 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1678301 T3296 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1
   [junit4]   2> 1678301 T3296 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1678302 T3296 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1678302 T3296 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1678303 T3296 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1/'
   [junit4]   2> 1678305 T3296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1/lib/classes/' to classloader
   [junit4]   2> 1678305 T3296 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1388276358524/collection1/lib/README' to classloader
   [junit4]   2> 1678349 T3296 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 1678390 T3296 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1678492 T3296 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1678514 T3296 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1679103 T3296 oass.IndexSchema.readSchema 

[...truncated too long message...]

lectionContext.runLeaderProcess(ElectionContext.java:258)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 1820340 T3382 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=27,cumulative_deletesById=13,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
   [junit4]   2> 1820341 T3382 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1820341 T3382 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1820341 T3382 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1820342 T3382 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1820343 T3382 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1820343 T3382 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7/index;done=false>>]
   [junit4]   2> 1820343 T3382 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7/index
   [junit4]   2> 1820344 T3382 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7;done=false>>]
   [junit4]   2> 1820344 T3382 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1388276348572/jetty7
   [junit4]   2> 1820344 T3382 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1820344 T3382 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1820345 T3382 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1820345 T3382 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fi, timezone=Etc/UCT
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=275901600,total=460980224
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexingPerformance, EnumFieldTest, TestFoldingMultitermQuery, TestQuerySenderNoQuery, SolrCmdDistributorTest, TestReload, BasicZkTest, UniqFieldsUpdateProcessorFactoryTest, BasicDistributedZk2Test, TestAtomicUpdateErrorCases, TestDistributedMissingSort, SolrIndexConfigTest, AddBlockUpdateTest, TestStressRecovery, CurrencyFieldOpenExchangeTest, TestInfoStreamLogging, TestDefaultSearchFieldResource, TestDistributedGrouping, DistributedQueryComponentCustomSortTest, TestSolrQueryParserDefaultOperatorResource, DirectUpdateHandlerTest, TestRangeQuery, TestMultiCoreConfBootstrap, TestCSVLoader, TestQuerySenderListener, ZkSolrClientTest, CachingDirectoryFactoryTest, SortByFunctionTest, TestComponentsName, SuggesterWFSTTest, ConnectionManagerTest, TestQueryTypes, TestFiltering, ConvertedLegacyTest, SuggestComponentTest, TestFieldTypeResource, FieldAnalysisRequestHandlerTest, OpenCloseCoreStressTest, MigrateRouteKeyTest, XsltUpdateRequestHandlerTest, TestSolrQueryParserResource, TestRecovery, TestFuzzyAnalyzedSuggestions, TestSolr4Spatial, BlockDirectoryTest, TestFastOutputStream, LeaderElectionIntegrationTest, RAMDirectoryFactoryTest, ZkNodePropsTest, NoCacheHeaderTest, TermVectorComponentDistributedTest, DisMaxRequestHandlerTest, TestSchemaResource, MultiTermTest, TestSolrXmlPersistence, RequestHandlersTest, HdfsChaosMonkeySafeLeaderTest, SolrCoreCheckLockOnStartupTest, DocumentAnalysisRequestHandlerTest, TestStressLucene, PrimUtilsTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, TestSolrJ, SpellCheckComponentTest, SolrPluginUtilsTest, TestHashPartitioner, SimplePostToolTest, TestIBSimilarityFactory, TestMaxScoreQueryParser, AliasIntegrationTest, TestDistribDocBasedVersion, PreAnalyzedUpdateProcessorTest, NumericFieldsTest, IndexSchemaRuntimeFieldTest, SpellingQueryConverterTest, TestLazyCores, TestSolrCoreProperties, TestSchemaVersionResource, TestCollapseQParserPlugin, TestStressReorder, TestFieldResource, OpenExchangeRatesOrgProviderTest, TestBinaryResponseWriter, MinimalSchemaTest, ClusterStateUpdateTest, TestRandomFaceting, TestFaceting, ZkControllerTest, TestRealTimeGet, HardAutoCommitTest, DistributedTermsComponentTest, StatsComponentTest, TestGroupingSearch, HighlighterTest, DistributedQueryElevationComponentTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SolrRequestParserTest, SuggesterTest, WordBreakSolrSpellCheckerTest, TestUpdate, TestWordDelimiterFilterFactory, DefaultValueUpdateProcessorTest, DistanceFunctionTest, CacheHeaderTest, TestSurroundQueryParser, DirectSolrSpellCheckerTest, FileBasedSpellCheckerTest, TestValueSourceCache, FastVectorHighlighterTest, IndexSchemaTest, TestCollationField, JSONWriterTest, MBeansHandlerTest, JsonLoaderTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, CoreContainerCoreInitFailuresTest, HighlighterConfigTest, AlternateDirectoryTest, TestSolrIndexConfig, ResponseLogComponentTest, SampleTest, TestConfig, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestXIncludeConfig, TestDFRSimilarityFactory, TestNumberUtils, TestLMDirichletSimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, ScriptEngineTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J1 in 155.36s, 1 test, 1 failure <<< FAILURES!

[...truncated 130 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:459: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1307: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:940: There were test failures: 348 suites, 1519 tests, 1 failure, 35 ignored (6 assumptions)

Total time: 60 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message