lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b117) - Build # 8550 - Still Failing!
Date Tue, 03 Dec 2013 01:45:34 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8550/
Java: 32bit/jdk1.8.0-ea-b117 -client -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=79 sum(shards)=80 cloudClient=80

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=79 sum(shards)=80 cloudClient=80
	at __randomizedtesting.SeedInfo.seed([E321068BBDFBE405:62C78893CAA48439]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1241)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:208)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:843)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10953 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 1105341 T3686 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1105346 T3686 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyNothingIsSafeTest-1386034785256
   [junit4]   2> 1105347 T3686 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1105348 T3687 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1105448 T3686 oasc.ZkTestServer.run start zk server on port:40574
   [junit4]   2> 1105449 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1105531 T3693 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@396f8b name:ZooKeeperConnection Watcher:127.0.0.1:40574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105532 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1105532 T3686 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1105536 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1105538 T3695 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@116a338 name:ZooKeeperConnection Watcher:127.0.0.1:40574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105538 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1105538 T3686 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1105544 T3686 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1105546 T3686 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1105548 T3686 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1105550 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1105551 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1105554 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1105555 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1105557 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1105558 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1105561 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1105561 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1105564 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1105565 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1105568 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1105569 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1105572 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1105572 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1105574 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1105575 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1105578 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1105578 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1105581 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1105582 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1105585 T3686 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1105586 T3686 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1105669 T3686 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1105676 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
   [junit4]   2> 1105681 T3686 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55478
   [junit4]   2> 1105682 T3686 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1105683 T3686 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1105683 T3686 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498
   [junit4]   2> 1105683 T3686 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/'
   [junit4]   2> 1105700 T3686 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/solr.xml
   [junit4]   2> 1105709 T3686 oasc.CoreContainer.<init> New CoreContainer 3104295
   [junit4]   2> 1105710 T3686 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/]
   [junit4]   2> 1105711 T3686 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1105711 T3686 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1105712 T3686 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1105712 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1105712 T3686 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1105713 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1105713 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1105714 T3686 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1105714 T3686 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1105717 T3686 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1105718 T3686 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1105718 T3686 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1105718 T3686 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40574/solr
   [junit4]   2> 1105719 T3686 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1105720 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1105721 T3706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f10a1 name:ZooKeeperConnection Watcher:127.0.0.1:40574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105721 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1105723 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1105724 T3708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71d988 name:ZooKeeperConnection Watcher:127.0.0.1:40574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1105724 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1105726 T3686 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1105729 T3686 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1105735 T3686 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1105737 T3686 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55478_
   [junit4]   2> 1105738 T3686 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55478_
   [junit4]   2> 1105740 T3686 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1105746 T3686 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1105748 T3686 oasc.Overseer.start Overseer (id=90835175686668291-127.0.0.1:55478_-n_0000000000) starting
   [junit4]   2> 1105753 T3686 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1105760 T3710 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1105761 T3686 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1105764 T3686 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1105766 T3686 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1105769 T3709 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1105770 T3711 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1105771 T3711 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1105774 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1105775 T3711 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1105776 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1105777 T3709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55478",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55478_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1105777 T3709 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1105779 T3709 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1105781 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1105782 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1106775 T3711 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1106776 T3711 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/collection1
   [junit4]   2> 1106776 T3711 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1106777 T3711 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1106777 T3711 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1106778 T3711 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/collection1/'
   [junit4]   2> 1106779 T3711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/collection1/lib/classes/' to classloader
   [junit4]   2> 1106779 T3711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/collection1/lib/README' to classloader
   [junit4]   2> 1106804 T3711 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1106820 T3711 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1106821 T3711 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1106829 T3711 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1106968 T3711 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1106970 T3711 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1106971 T3711 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1106973 T3711 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1106992 T3711 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1106992 T3711 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1386034785498/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/control/data/
   [junit4]   2> 1106993 T3711 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b8420d
   [junit4]   2> 1106993 T3711 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/control/data
   [junit4]   2> 1106994 T3711 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/control/data/index/
   [junit4]   2> 1106994 T3711 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1106994 T3711 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/control/data/index
   [junit4]   2> 1106995 T3711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=38.45703125, floorSegmentMB=1.552734375, forceMergeDeletesPctAllowed=8.910029840883553, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1106998 T3711 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1106999 T3711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1107001 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1107002 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1107002 T3711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1107002 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1107003 T3711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1107003 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1107003 T3711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1107003 T3711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1107004 T3711 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1107004 T3711 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1107005 T3711 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1107005 T3711 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1107005 T3711 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1107006 T3711 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1107006 T3711 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1107006 T3711 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1107007 T3711 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1107015 T3711 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1107016 T3711 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1107017 T3711 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1107018 T3711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8337330170354337]
   [junit4]   2> 1107019 T3711 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1107019 T3711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1107019 T3711 oass.SolrIndexSearcher.<init> Opening Searcher@8a26df main
   [junit4]   2> 1107022 T3712 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8a26df main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1107023 T3711 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1107024 T3711 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55478 collection:control_collection shard:shard1
   [junit4]   2> 1107024 T3711 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1107031 T3711 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1107033 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107034 T3711 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1107034 T3711 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8021 name=collection1 org.apache.solr.core.SolrCore@46b33 url=http://127.0.0.1:55478/collection1 node=127.0.0.1:55478_ C8021_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:55478, core=collection1, node_name=127.0.0.1:55478_}
   [junit4]   2> 1107034 T3711 C8021 P55478 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55478/collection1/
   [junit4]   2> 1107035 T3711 C8021 P55478 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1107035 T3711 C8021 P55478 oasc.SyncStrategy.syncToMe http://127.0.0.1:55478/collection1/ has no replicas
   [junit4]   2> 1107035 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1107035 T3711 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55478/collection1/ shard1
   [junit4]   2> 1107036 T3711 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1107038 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107041 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107044 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107146 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1107192 T3711 oasc.ZkController.register We are http://127.0.0.1:55478/collection1/ and leader is http://127.0.0.1:55478/collection1/
   [junit4]   2> 1107193 T3711 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55478
   [junit4]   2> 1107193 T3711 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1107193 T3711 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1107193 T3711 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1107194 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107195 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107195 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107195 T3711 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1107197 T3686 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1107197 T3686 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1107199 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107200 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1107201 T3715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1887306 name:ZooKeeperConnection Watcher:127.0.0.1:40574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107202 T3709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55478",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55478_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1107202 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107203 T3686 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1107205 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107206 T3686 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1107307 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1107307 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1107307 T3686 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1107311 T3686 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46582
   [junit4]   2> 1107312 T3686 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1107312 T3686 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1107313 T3686 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116
   [junit4]   2> 1107313 T3686 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/'
   [junit4]   2> 1107335 T3686 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/solr.xml
   [junit4]   2> 1107348 T3686 oasc.CoreContainer.<init> New CoreContainer 9640176
   [junit4]   2> 1107349 T3686 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/]
   [junit4]   2> 1107350 T3686 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1107351 T3686 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1107351 T3686 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1107352 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1107352 T3686 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1107352 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1107353 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1107353 T3686 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1107354 T3686 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1107369 T3686 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1107369 T3686 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1107370 T3686 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1107371 T3686 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40574/solr
   [junit4]   2> 1107371 T3686 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1107372 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107375 T3726 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9efb0a name:ZooKeeperConnection Watcher:127.0.0.1:40574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107375 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107379 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107380 T3728 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1624253 name:ZooKeeperConnection Watcher:127.0.0.1:40574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107380 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107386 T3686 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1108390 T3686 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46582_
   [junit4]   2> 1108392 T3686 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46582_
   [junit4]   2> 1108395 T3715 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1108395 T3708 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1108395 T3728 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1108400 T3729 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1108401 T3729 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1108402 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1108403 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1108403 T3729 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1108403 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1108405 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1108405 T3709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46582",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46582_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1108406 T3709 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1108406 T3709 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1108408 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1108511 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1108511 T3728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1108511 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1109404 T3729 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1109404 T3729 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/collection1
   [junit4]   2> 1109404 T3729 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1109405 T3729 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1109405 T3729 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1109407 T3729 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/collection1/'
   [junit4]   2> 1109408 T3729 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/collection1/lib/classes/' to classloader
   [junit4]   2> 1109408 T3729 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/collection1/lib/README' to classloader
   [junit4]   2> 1109435 T3729 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1109448 T3729 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1109449 T3729 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1109456 T3729 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1109611 T3729 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1109613 T3729 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1109614 T3729 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1109617 T3729 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1109641 T3729 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1109642 T3729 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1386034787116/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty1/
   [junit4]   2> 1109642 T3729 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b8420d
   [junit4]   2> 1109643 T3729 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty1
   [junit4]   2> 1109643 T3729 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty1/index/
   [junit4]   2> 1109643 T3729 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1109644 T3729 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty1/index
   [junit4]   2> 1109645 T3729 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=38.45703125, floorSegmentMB=1.552734375, forceMergeDeletesPctAllowed=8.910029840883553, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1109647 T3729 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1109647 T3729 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1109653 T3729 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1109654 T3729 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1109654 T3729 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1109654 T3729 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1109655 T3729 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1109655 T3729 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1109656 T3729 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1109656 T3729 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1109657 T3729 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1109658 T3729 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1109659 T3729 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1109659 T3729 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1109660 T3729 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1109661 T3729 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1109661 T3729 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1109662 T3729 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1109663 T3729 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1109666 T3729 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1109672 T3729 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1109673 T3729 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1109674 T3729 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8337330170354337]
   [junit4]   2> 1109675 T3729 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1109676 T3729 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1109676 T3729 oass.SolrIndexSearcher.<init> Opening Searcher@6e9545 main
   [junit4]   2> 1109681 T3730 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e9545 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1109686 T3729 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1109686 T3729 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46582 collection:collection1 shard:shard2
   [junit4]   2> 1109687 T3729 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1109694 T3729 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1109696 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109697 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109697 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109697 T3729 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1109697 T3729 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8022 name=collection1 org.apache.solr.core.SolrCore@40e3e8 url=http://127.0.0.1:46582/collection1 node=127.0.0.1:46582_ C8022_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:46582, core=collection1, node_name=127.0.0.1:46582_}
   [junit4]   2> 1109698 T3729 C8022 P46582 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46582/collection1/
   [junit4]   2> 1109698 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1109698 T3729 C8022 P46582 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1109699 T3729 C8022 P46582 oasc.SyncStrategy.syncToMe http://127.0.0.1:46582/collection1/ has no replicas
   [junit4]   2> 1109699 T3729 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46582/collection1/ shard2
   [junit4]   2> 1109699 T3729 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1109700 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109707 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109711 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109813 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1109813 T3728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1109813 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1109859 T3729 oasc.ZkController.register We are http://127.0.0.1:46582/collection1/ and leader is http://127.0.0.1:46582/collection1/
   [junit4]   2> 1109860 T3729 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46582
   [junit4]   2> 1109860 T3729 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1109860 T3729 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1109860 T3729 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1109862 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109862 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109863 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109863 T3729 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1109864 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1109864 T3686 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1109865 T3686 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1109865 T3709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46582",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46582_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1109868 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109944 T3686 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1109946 T3686 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35774
   [junit4]   2> 1109947 T3686 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1109947 T3686 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1109948 T3686 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775
   [junit4]   2> 1109948 T3686 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/'
   [junit4]   2> 1109968 T3686 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/solr.xml
   [junit4]   2> 1109971 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1109971 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1109971 T3728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1109978 T3686 oasc.CoreContainer.<init> New CoreContainer 13870989
   [junit4]   2> 1109979 T3686 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/]
   [junit4]   2> 1109980 T3686 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1109980 T3686 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1109980 T3686 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1109981 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1109981 T3686 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1109982 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1109982 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1109982 T3686 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1109983 T3686 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1109987 T3686 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1109987 T3686 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1109987 T3686 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1109988 T3686 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40574/solr
   [junit4]   2> 1109988 T3686 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1109989 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1109990 T3742 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@199c49a name:ZooKeeperConnection Watcher:127.0.0.1:40574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1109991 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1109995 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1109997 T3744 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a8980e name:ZooKeeperConnection Watcher:127.0.0.1:40574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1109997 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1110002 T3686 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111006 T3686 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35774_
   [junit4]   2> 1111007 T3686 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35774_
   [junit4]   2> 1111010 T3728 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111010 T3744 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111010 T3715 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111010 T3708 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111014 T3745 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1111014 T3745 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1111015 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1111015 T3745 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1111016 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1111016 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1111018 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1111019 T3709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35774",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35774_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1111019 T3709 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1111019 T3709 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1111022 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1111125 T3744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1111125 T3728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1111125 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1111125 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112016 T3745 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1112017 T3745 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/collection1
   [junit4]   2> 1112017 T3745 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1112018 T3745 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1112018 T3745 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1112019 T3745 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/collection1/'
   [junit4]   2> 1112020 T3745 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/collection1/lib/classes/' to classloader
   [junit4]   2> 1112021 T3745 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/collection1/lib/README' to classloader
   [junit4]   2> 1112048 T3745 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1112065 T3745 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1112066 T3745 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1112074 T3745 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1112220 T3745 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1112222 T3745 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1112223 T3745 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1112228 T3745 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1112248 T3745 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1112249 T3745 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1386034789775/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty2/
   [junit4]   2> 1112249 T3745 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b8420d
   [junit4]   2> 1112249 T3745 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty2
   [junit4]   2> 1112250 T3745 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty2/index/
   [junit4]   2> 1112250 T3745 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1112251 T3745 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty2/index
   [junit4]   2> 1112251 T3745 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=38.45703125, floorSegmentMB=1.552734375, forceMergeDeletesPctAllowed=8.910029840883553, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1112253 T3745 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1112254 T3745 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1112258 T3745 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1112259 T3745 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1112260 T3745 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1112260 T3745 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1112260 T3745 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1112260 T3745 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1112261 T3745 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1112261 T3745 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1112262 T3745 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1112263 T3745 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1112263 T3745 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1112264 T3745 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1112264 T3745 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1112265 T3745 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1112265 T3745 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1112266 T3745 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1112267 T3745 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1112269 T3745 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1112270 T3745 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1112271 T3745 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1112272 T3745 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=31, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8337330170354337]
   [junit4]   2> 1112273 T3745 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1112273 T3745 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1112273 T3745 oass.SolrIndexSearcher.<init> Opening Searcher@cbd4d2 main
   [junit4]   2> 1112278 T3746 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@cbd4d2 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1112283 T3745 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1112283 T3745 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35774 collection:collection1 shard:shard1
   [junit4]   2> 1112284 T3745 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1112292 T3745 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1112294 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112294 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112294 T3745 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1112295 T3745 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1112294 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C8023 name=collection1 org.apache.solr.core.SolrCore@1ed3410 url=http://127.0.0.1:35774/collection1 node=127.0.0.1:35774_ C8023_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35774, core=collection1, node_name=127.0.0.1:35774_}
   [junit4]   2> 1112295 T3745 C8023 P35774 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35774/collection1/
   [junit4]   2> 1112296 T3745 C8023 P35774 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1112296 T3745 C8023 P35774 oasc.SyncStrategy.syncToMe http://127.0.0.1:35774/collection1/ has no replicas
   [junit4]   2> 1112296 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1112296 T3745 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35774/collection1/ shard1
   [junit4]   2> 1112297 T3745 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1112299 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112303 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112308 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112410 T3744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112410 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112410 T3728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112410 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112454 T3745 oasc.ZkController.register We are http://127.0.0.1:35774/collection1/ and leader is http://127.0.0.1:35774/collection1/
   [junit4]   2> 1112455 T3745 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35774
   [junit4]   2> 1112455 T3745 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1112455 T3745 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1112455 T3745 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1112457 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112457 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112457 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112458 T3745 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1112459 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1112459 T3686 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1112460 T3686 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1112460 T3709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35774",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35774_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1112463 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1112568 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112568 T3744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112568 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112568 T3728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1112577 T3686 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1112579 T3686 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45983
   [junit4]   2> 1112586 T3686 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1112587 T3686 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1112587 T3686 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371
   [junit4]   2> 1112588 T3686 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/'
   [junit4]   2> 1112614 T3686 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/solr.xml
   [junit4]   2> 1112624 T3686 oasc.CoreContainer.<init> New CoreContainer 32550907
   [junit4]   2> 1112625 T3686 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/]
   [junit4]   2> 1112626 T3686 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1112626 T3686 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1112626 T3686 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1112627 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1112627 T3686 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1112628 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1112628 T3686 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1112629 T3686 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1112629 T3686 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1112632 T3686 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1112633 T3686 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1112633 T3686 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1112633 T3686 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40574/solr
   [junit4]   2> 1112634 T3686 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1112635 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112636 T3758 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bdbff name:ZooKeeperConnection Watcher:127.0.0.1:40574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112637 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112640 T3686 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112641 T3760 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9f7ef0 name:ZooKeeperConnection Watcher:127.0.0.1:40574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112641 T3686 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112645 T3686 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1113648 T3686 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45983_
   [junit4]   2> 1113649 T3686 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45983_
   [junit4]   2> 1113652 T3715 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1113653 T3708 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1113653 T3728 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1113653 T3760 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1113653 T3744 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1113658 T3761 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1113658 T3761 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1113659 T3761 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1113660 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1113660 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1113660 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1113661 T3709 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1113662 T3709 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45983",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45983_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1113663 T3709 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1113663 T3709 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1113668 T3708 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1113770 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1113770 T3708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1113770 T3760 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1113770 T3728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1113770 T3744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1114660 T3761 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1114660 T3761 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/collection1
   [junit4]   2> 1114661 T3761 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1114661 T3761 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1114662 T3761 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1114663 T3761 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/collection1/'
   [junit4]   2> 1114663 T3761 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/collection1/lib/classes/' to classloader
   [junit4]   2> 1114664 T3761 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/collection1/lib/README' to classloader
   [junit4]   2> 1114692 T3761 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1114711 T3761 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1114713 T3761 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1114724 T3761 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1114870 T3761 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1114872 T3761 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1114873 T3761 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1114876 T3761 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1114900 T3761 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1114901 T3761 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1386034792371/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty3/
   [junit4]   2> 1114901 T3761 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b8420d
   [junit4]   2> 1114902 T3761 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty3
   [junit4]   2> 1114902 T3761 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty3/index/
   [junit4]   2> 1114903 T3761 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 1114904 T3761 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty3/index
   [junit4]   2> 1114905 T3761 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=15, maxMergedSegmentMB=38.45703125, floorSegmentMB=1.552734375, forceMergeDeletesPctAllowed=8.910029840883553, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1114907 T3761 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 1114907 T3761 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1114911 T3761 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1114911 T3761 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1114911 T3761 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1114912 T3761 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1114912 T3761 oasup.UpdateRequestProcessorChain.

[...truncated too long message...]

ributedSearchTestCase.java:843)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 1156576 T3686 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> ASYNC  NEW_CORE C8084 name=collection1 org.apache.solr.core.SolrCore@1ebebaf url=http://127.0.0.1:57813/collection1 node=127.0.0.1:57813_ C8084_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:57813, core=collection1, node_name=127.0.0.1:57813_}
   [junit4]   2> 1156660 T3830 C8084 P57813 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 1156661 T3830 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 1156661 T3830 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1ebebaf
   [junit4]   2> 1156663 T3830 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=72,cumulative_deletesById=37,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
   [junit4]   2> 1156663 T3830 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1156663 T3830 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1156663 T3830 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1156664 T3830 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1156664 T3830 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1156664 T3830 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty7 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty7;done=false>>]
   [junit4]   2> 1156665 T3830 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty7
   [junit4]   2> 1156665 T3830 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty7/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty7/index;done=false>>]
   [junit4]   2> 1156665 T3830 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1386034785256/jetty7/index
   [junit4]   2> 1156665 T3830 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1156665 T3830 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=sk_SK, timezone=America/Indiana/Vincennes
   [junit4]   2> NOTE: Linux 3.8.0-33-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=100985328,total=236433408
   [junit4]   2> NOTE: All tests run in this JVM: [FieldFacetTest, TestAnalyzeInfixSuggestions, TestNonNRTOpen, CacheHeaderTest, TestModifyConfFiles, TestLMJelinekMercerSimilarityFactory, SimpleFacetsTest, AssignTest, DistributedSpellCheckComponentTest, UpdateParamsTest, TestPHPSerializedResponseWriter, TestSurroundQueryParser, SliceStateTest, SampleTest, TestSuggestSpellingConverter, RecoveryZkTest, SpatialFilterTest, DistributedQueryElevationComponentTest, TestNRTOpen, TestFaceting, QueryEqualityTest, EchoParamsTest, CustomCollectionTest, TestSort, AutoCommitTest, TestCollationFieldDocValues, TestPseudoReturnFields, TestGroupingSearch, TestTrie, BadCopyFieldTest, ParsingFieldUpdateProcessorsTest, TestJmxMonitoredMap, TestRandomMergePolicy, TestCloudManagedSchemaAddField, MoreLikeThisHandlerTest, RangeFacetTest, CoreMergeIndexesAdminHandlerTest, TestValueSourceCache, WordBreakSolrSpellCheckerTest, TestLRUCache, TestConfig, SystemInfoHandlerTest, TestFieldTypeResource, TestDocBasedVersionConstraints, TestSolrDeletionPolicy1, TestWriterPerf, ClusterStateUpdateTest, ResponseLogComponentTest, TestSolrXml, SuggesterTest, FieldFacetExtrasTest, SearchHandlerTest, ZkCLITest, PreAnalyzedUpdateProcessorTest, CoreAdminCreateDiscoverTest, AbstractAnalyticsFacetTest, TermsComponentTest, TestSolrIndexConfig, AddBlockUpdateTest, TestBM25SimilarityFactory, TestFieldCollectionResource, OverseerTest, TestDocSet, StressHdfsTest, PeerSyncTest, TestLMDirichletSimilarityFactory, TestPostingsSolrHighlighter, TestMergePolicyConfig, ResourceLoaderTest, TestReversedWildcardFilterFactory, TestDocumentBuilder, CircularListTest, SpellPossibilityIteratorTest, UnloadDistributedZkTest, ZkControllerTest, TestSchemaVersionResource, ExternalFileFieldSortTest, QueryFacetTest, SchemaVersionSpecificBehaviorTest, TestCoreContainer, TestRemoteStreaming, PolyFieldTest, FastVectorHighlighterTest, URLClassifyProcessorTest, IndexSchemaTest, HdfsBasicDistributedZkTest, StatelessScriptUpdateProcessorFactoryTest, SyncSliceTest, OpenCloseCoreStressTest, BasicZkTest, TestRandomFaceting, ShardRoutingCustomTest, SliceStateUpdateTest, TestDistributedGrouping, TestHashPartitioner, TermVectorComponentDistributedTest, TestRealTimeGet, TestReload, TestStressVersions, HardAutoCommitTest, DistributedTermsComponentTest, TestSolr4Spatial, SolrCoreTest, SpellCheckComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, TestFiltering, CurrencyFieldOpenExchangeTest, AnalysisAfterCoreReloadTest, SuggesterFSTTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, SuggesterWFSTTest, TestCSVLoader, SolrCoreCheckLockOnStartupTest, TestWordDelimiterFilterFactory, DocValuesMultiTest, SolrInfoMBeanTest, LukeRequestHandlerTest, PrimitiveFieldTypeTest, TestOmitPositions, XmlUpdateRequestHandlerTest, DocumentBuilderTest, FieldAnalysisRequestHandlerTest, TestArbitraryIndexDir, LoggingHandlerTest, RegexBoostProcessorTest, ReturnFieldsTest, JSONWriterTest, QueryParsingTest, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, CoreContainerCoreInitFailuresTest, TestBinaryResponseWriter, SOLR749Test, CopyFieldTest, BadComponentTest, TestSearchPerf, NumericFieldsTest, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestNumberUtils, LegacyHTMLStripCharFilterTest, TestFastOutputStream, ScriptEngineTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J0 in 52.27s, 1 test, 1 failure <<< FAILURES!

[...truncated 149 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:420: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:400: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:909: There were test failures: 357 suites, 1570 tests, 1 failure, 34 ignored (6 assumptions)

Total time: 48 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b117 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message