lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1062 - Failure!
Date Tue, 26 Nov 2013 21:42:28 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1062/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=70 sum(shards)=71 cloudClient=71

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=70 sum(shards)=71 cloudClient=71
	at __randomizedtesting.SeedInfo.seed([BE49E0FA126C0054:3FAF6EE265336068]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1242)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:197)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10397 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 1414487 T5208 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1414494 T5208 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyNothingIsSafeTest-1385500516177
   [junit4]   2> 1414495 T5208 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1414496 T5209 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1414600 T5208 oasc.ZkTestServer.run start zk server on port:50524
   [junit4]   2> 1414602 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1414610 T5215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5eb4e65 name:ZooKeeperConnection Watcher:127.0.0.1:50524 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1414611 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1414611 T5208 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1414625 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1414630 T5217 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3597775f name:ZooKeeperConnection Watcher:127.0.0.1:50524/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1414630 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1414630 T5208 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1414638 T5208 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1414646 T5208 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1414653 T5208 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1414662 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1414663 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1414678 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1414679 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1414687 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1414688 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1414695 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1414695 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1414703 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1414704 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1414711 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1414712 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1414718 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1414719 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1414725 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1414726 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1414734 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1414735 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1414742 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1414742 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1414752 T5208 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1414753 T5208 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1415053 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1415068 T5208 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50527
   [junit4]   2> 1415069 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1415070 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1415070 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446
   [junit4]   2> 1415071 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/'
   [junit4]   2> 1415099 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/solr.xml
   [junit4]   2> 1415163 T5208 oasc.CoreContainer.<init> New CoreContainer 532188197
   [junit4]   2> 1415163 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/]
   [junit4]   2> 1415164 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1415165 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1415165 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1415165 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1415166 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1415166 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1415166 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1415166 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1415167 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1415180 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1415181 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1415181 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1415181 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50524/solr
   [junit4]   2> 1415181 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1415183 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1415187 T5228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fe0fa5d name:ZooKeeperConnection Watcher:127.0.0.1:50524 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1415187 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1415192 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1415196 T5230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@289422f9 name:ZooKeeperConnection Watcher:127.0.0.1:50524/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1415196 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1415200 T5208 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1415209 T5208 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1415217 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1415222 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50527_
   [junit4]   2> 1415224 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50527_
   [junit4]   2> 1415231 T5208 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1415243 T5208 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1415249 T5208 oasc.Overseer.start Overseer (id=90800161828569091-127.0.0.1:50527_-n_0000000000) starting
   [junit4]   2> 1415256 T5208 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1415267 T5232 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1415268 T5208 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1415275 T5208 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1415279 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1415288 T5231 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1415295 T5233 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1415296 T5233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1415298 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1415299 T5233 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1415302 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1415304 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50527",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50527_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1415304 T5231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1415304 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1415311 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1415312 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1416300 T5233 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1416301 T5233 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/collection1
   [junit4]   2> 1416301 T5233 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1416303 T5233 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1416303 T5233 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1416306 T5233 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/collection1/'
   [junit4]   2> 1416308 T5233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/collection1/lib/classes/' to classloader
   [junit4]   2> 1416308 T5233 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/collection1/lib/README' to classloader
   [junit4]   2> 1416369 T5233 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1416433 T5233 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1416436 T5233 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1416445 T5233 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1416978 T5233 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1416987 T5233 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1416992 T5233 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1417022 T5233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1417028 T5233 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1417034 T5233 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1417037 T5233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1417038 T5233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1417038 T5233 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1417041 T5233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1417041 T5233 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1417042 T5233 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1417042 T5233 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1385500516446/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/control/data/
   [junit4]   2> 1417042 T5233 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff4569
   [junit4]   2> 1417043 T5233 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/control/data
   [junit4]   2> 1417044 T5233 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/control/data/index/
   [junit4]   2> 1417044 T5233 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1417045 T5233 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/control/data/index
   [junit4]   2> 1417045 T5233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5ecebd45
   [junit4]   2> 1417049 T5233 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1417049 T5233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1417053 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1417054 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1417054 T5233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1417054 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1417054 T5233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1417055 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1417055 T5233 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1417056 T5233 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1417056 T5233 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1417056 T5233 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1417057 T5233 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1417057 T5233 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1417058 T5233 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1417058 T5233 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1417058 T5233 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1417059 T5233 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1417068 T5233 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1417072 T5233 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1417072 T5233 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1417073 T5233 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7911701994712995]
   [junit4]   2> 1417075 T5233 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1417076 T5233 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1417076 T5233 oass.SolrIndexSearcher.<init> Opening Searcher@537806c main
   [junit4]   2> 1417079 T5234 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@537806c main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1417080 T5233 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1417080 T5233 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50527 collection:control_collection shard:shard1
   [junit4]   2> 1417082 T5233 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1417100 T5233 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1417105 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417107 T5233 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1417108 T5233 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3151 name=collection1 org.apache.solr.core.SolrCore@27e2ef7e url=http://127.0.0.1:50527/collection1 node=127.0.0.1:50527_ C3151_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:50527, core=collection1, node_name=127.0.0.1:50527_}
   [junit4]   2> 1417108 T5233 C3151 P50527 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50527/collection1/
   [junit4]   2> 1417108 T5233 C3151 P50527 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1417109 T5233 C3151 P50527 oasc.SyncStrategy.syncToMe http://127.0.0.1:50527/collection1/ has no replicas
   [junit4]   2> 1417109 T5233 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50527/collection1/ shard1
   [junit4]   2> 1417109 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1417109 T5233 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1417115 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417117 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1417127 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1417133 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417239 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1417279 T5233 oasc.ZkController.register We are http://127.0.0.1:50527/collection1/ and leader is http://127.0.0.1:50527/collection1/
   [junit4]   2> 1417279 T5233 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50527
   [junit4]   2> 1417279 T5233 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1417279 T5233 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1417280 T5233 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1417282 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417282 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417283 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417283 T5233 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1417286 T5208 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1417286 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1417287 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1417299 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50527",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50527_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1417300 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1417304 T5237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bb18972 name:ZooKeeperConnection Watcher:127.0.0.1:50524/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1417305 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1417306 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1417307 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1417314 T5208 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1417411 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1417411 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1417611 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1417614 T5208 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50531
   [junit4]   2> 1417616 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1417617 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1417617 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997
   [junit4]   2> 1417617 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/'
   [junit4]   2> 1417644 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/solr.xml
   [junit4]   2> 1417713 T5208 oasc.CoreContainer.<init> New CoreContainer 997316304
   [junit4]   2> 1417714 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/]
   [junit4]   2> 1417715 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1417716 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1417716 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1417716 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1417716 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1417717 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1417717 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1417717 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1417718 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1417730 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1417731 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1417731 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1417732 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50524/solr
   [junit4]   2> 1417732 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1417733 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1417738 T5248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd9568c name:ZooKeeperConnection Watcher:127.0.0.1:50524 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1417739 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1417743 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1417747 T5250 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e5198f7 name:ZooKeeperConnection Watcher:127.0.0.1:50524/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1417748 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1417756 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1418765 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50531_
   [junit4]   2> 1418768 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50531_
   [junit4]   2> 1418775 T5250 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1418776 T5237 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1418776 T5230 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1418791 T5251 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1418792 T5251 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1418795 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418795 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418795 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418795 T5251 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1418799 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1418801 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50531",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50531_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1418801 T5231 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1418801 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1418807 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1418915 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1418915 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1418915 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1419797 T5251 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1419797 T5251 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/collection1
   [junit4]   2> 1419798 T5251 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1419799 T5251 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1419799 T5251 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1419802 T5251 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/collection1/'
   [junit4]   2> 1419803 T5251 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/collection1/lib/classes/' to classloader
   [junit4]   2> 1419804 T5251 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/collection1/lib/README' to classloader
   [junit4]   2> 1419859 T5251 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1419922 T5251 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1419925 T5251 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1419934 T5251 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1420452 T5251 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1420461 T5251 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1420464 T5251 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1420492 T5251 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1420499 T5251 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1420504 T5251 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1420507 T5251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1420507 T5251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1420508 T5251 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1420511 T5251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1420512 T5251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1420512 T5251 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1420512 T5251 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1385500518997/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty1/
   [junit4]   2> 1420513 T5251 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff4569
   [junit4]   2> 1420514 T5251 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty1
   [junit4]   2> 1420514 T5251 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty1/index/
   [junit4]   2> 1420515 T5251 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1420516 T5251 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty1/index
   [junit4]   2> 1420516 T5251 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7b832128
   [junit4]   2> 1420519 T5251 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1420520 T5251 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1420525 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1420525 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1420525 T5251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1420526 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1420526 T5251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1420526 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1420527 T5251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1420527 T5251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1420527 T5251 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1420528 T5251 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1420529 T5251 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1420529 T5251 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1420529 T5251 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1420530 T5251 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1420530 T5251 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1420531 T5251 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1420540 T5251 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1420555 T5251 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1420555 T5251 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1420556 T5251 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7911701994712995]
   [junit4]   2> 1420557 T5251 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1420558 T5251 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1420559 T5251 oass.SolrIndexSearcher.<init> Opening Searcher@5845a129 main
   [junit4]   2> 1420570 T5252 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5845a129 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1420573 T5251 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1420573 T5251 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50531 collection:collection1 shard:shard1
   [junit4]   2> 1420576 T5251 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1420596 T5251 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1420600 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420600 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420601 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420602 T5251 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1420602 T5251 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3152 name=collection1 org.apache.solr.core.SolrCore@7bde4dd1 url=http://127.0.0.1:50531/collection1 node=127.0.0.1:50531_ C3152_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50531, core=collection1, node_name=127.0.0.1:50531_}
   [junit4]   2> 1420603 T5251 C3152 P50531 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50531/collection1/
   [junit4]   2> 1420603 T5251 C3152 P50531 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1420603 T5251 C3152 P50531 oasc.SyncStrategy.syncToMe http://127.0.0.1:50531/collection1/ has no replicas
   [junit4]   2> 1420604 T5251 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50531/collection1/ shard1
   [junit4]   2> 1420604 T5251 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1420605 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1420611 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420613 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420614 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420614 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420623 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1420631 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420739 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420740 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420739 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420777 T5251 oasc.ZkController.register We are http://127.0.0.1:50531/collection1/ and leader is http://127.0.0.1:50531/collection1/
   [junit4]   2> 1420777 T5251 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50531
   [junit4]   2> 1420777 T5251 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1420777 T5251 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1420778 T5251 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1420780 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420781 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420781 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420781 T5251 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1420783 T5208 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1420784 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1420784 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1420786 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50531",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50531_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1420792 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1420896 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420896 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1420903 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1421080 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1421085 T5208 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50534
   [junit4]   2> 1421086 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1421087 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1421087 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468
   [junit4]   2> 1421087 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/'
   [junit4]   2> 1421114 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/solr.xml
   [junit4]   2> 1421178 T5208 oasc.CoreContainer.<init> New CoreContainer 2089941493
   [junit4]   2> 1421178 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/]
   [junit4]   2> 1421180 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1421181 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1421181 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1421181 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1421181 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1421182 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1421182 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1421182 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1421183 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1421196 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1421196 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1421197 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1421197 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50524/solr
   [junit4]   2> 1421197 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1421199 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1421202 T5264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a1200ad name:ZooKeeperConnection Watcher:127.0.0.1:50524 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1421203 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1421208 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1421211 T5266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aec6318 name:ZooKeeperConnection Watcher:127.0.0.1:50524/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1421212 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1421223 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1422233 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50534_
   [junit4]   2> 1422235 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50534_
   [junit4]   2> 1422241 T5266 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1422241 T5250 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1422241 T5237 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1422242 T5230 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1422254 T5267 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1422254 T5267 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1422256 T5267 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1422256 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1422257 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1422257 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1422261 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1422263 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50534",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50534_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1422263 T5231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1422264 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1422269 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1422378 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1422378 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1422378 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1422378 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1423258 T5267 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1423258 T5267 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/collection1
   [junit4]   2> 1423258 T5267 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1423260 T5267 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1423260 T5267 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1423263 T5267 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/collection1/'
   [junit4]   2> 1423264 T5267 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/collection1/lib/classes/' to classloader
   [junit4]   2> 1423264 T5267 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/collection1/lib/README' to classloader
   [junit4]   2> 1423322 T5267 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1423393 T5267 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1423397 T5267 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1423407 T5267 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1423947 T5267 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1423956 T5267 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1423960 T5267 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1423991 T5267 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1424008 T5267 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1424015 T5267 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1424019 T5267 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1424019 T5267 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1424019 T5267 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1424023 T5267 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1424023 T5267 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1424023 T5267 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1424024 T5267 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1385500522468/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty2/
   [junit4]   2> 1424024 T5267 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff4569
   [junit4]   2> 1424025 T5267 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty2
   [junit4]   2> 1424026 T5267 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty2/index/
   [junit4]   2> 1424026 T5267 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1424027 T5267 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty2/index
   [junit4]   2> 1424027 T5267 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@63ba7035
   [junit4]   2> 1424030 T5267 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1424030 T5267 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1424035 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1424036 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1424036 T5267 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1424036 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1424036 T5267 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1424037 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1424037 T5267 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1424037 T5267 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1424038 T5267 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1424038 T5267 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1424039 T5267 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1424039 T5267 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1424039 T5267 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1424040 T5267 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1424040 T5267 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1424041 T5267 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1424050 T5267 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1424056 T5267 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1424057 T5267 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1424059 T5267 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=46, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7911701994712995]
   [junit4]   2> 1424061 T5267 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1385500516177/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1424061 T5267 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1424061 T5267 oass.SolrIndexSearcher.<init> Opening Searcher@2df8ffd1 main
   [junit4]   2> 1424520 T5268 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2df8ffd1 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1424526 T5267 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1424527 T5267 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50534 collection:collection1 shard:shard2
   [junit4]   2> 1424530 T5267 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1424561 T5267 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1424573 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424573 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424574 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424575 T5267 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1424575 T5267 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3153 name=collection1 org.apache.solr.core.SolrCore@72723875 url=http://127.0.0.1:50534/collection1 node=127.0.0.1:50534_ C3153_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50534, core=collection1, node_name=127.0.0.1:50534_}
   [junit4]   2> 1424576 T5267 C3153 P50534 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50534/collection1/
   [junit4]   2> 1424577 T5267 C3153 P50534 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1424578 T5267 C3153 P50534 oasc.SyncStrategy.syncToMe http://127.0.0.1:50534/collection1/ has no replicas
   [junit4]   2> 1424578 T5267 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50534/collection1/ shard2
   [junit4]   2> 1424579 T5267 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1424580 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1424595 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424599 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424599 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424600 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424600 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424611 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1424619 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424725 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424725 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424725 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424725 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424756 T5267 oasc.ZkController.register We are http://127.0.0.1:50534/collection1/ and leader is http://127.0.0.1:50534/collection1/
   [junit4]   2> 1424756 T5267 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50534
   [junit4]   2> 1424757 T5267 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1424757 T5267 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1424757 T5267 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1424760 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424760 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424760 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424762 T5267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1424764 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1424765 T5208 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1424765 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1424766 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50534",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50534_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1424772 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1424875 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424876 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424876 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1424884 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1425063 T5208 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1425068 T5208 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50537
   [junit4]   2> 1425070 T5208 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1425070 T5208 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1425070 T5208 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1385500526449
   [junit4]   2> 1425071 T5208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1385500526449/'
   [junit4]   2> 1425097 T5208 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1385500526449/solr.xml
   [junit4]   2> 1425163 T5208 oasc.CoreContainer.<init> New CoreContainer 2021952710
   [junit4]   2> 1425164 T5208 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1385500526449/]
   [junit4]   2> 1425166 T5208 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1425166 T5208 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1425167 T5208 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1425167 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1425167 T5208 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1425168 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1425168 T5208 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1425168 T5208 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1425168 T5208 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1425181 T5208 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1425182 T5208 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1425182 T5208 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1425182 T5208 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50524/solr
   [junit4]   2> 1425183 T5208 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1425184 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1425188 T5280 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e0c40db name:ZooKeeperConnection Watcher:127.0.0.1:50524 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1425190 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1425194 T5208 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1425198 T5282 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e764ea1 name:ZooKeeperConnection Watcher:127.0.0.1:50524/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1425198 T5208 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1425208 T5208 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1426218 T5208 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50537_
   [junit4]   2> 1426221 T5208 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50537_
   [junit4]   2> 1426228 T5230 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1426229 T5282 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1426229 T5237 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1426229 T5250 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1426230 T5266 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1426244 T5283 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1426245 T5283 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1426247 T5283 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1426247 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1426248 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1426248 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1426251 T5231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1426252 T5231 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50537",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50537_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1426252 T5231 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1426253 T5231 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1426259 T5230 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1426365 T5230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1426365 T5237 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1426365 T5266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1426365 T5282 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1426365 T5250 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1427248 T5283 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1427249 T5283 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1385500526449/collection1
   [junit4]   2> 1427249 T5283 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1427250 T5283 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1427251 T5283 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1427253 T5283 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1385500526449/collection1/'
   [junit4]   2> 1427254 T5283 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1385500526449/collection1/lib/classes/' to classloader
   [junit4]   2> 1427255 T5283 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apach

[...truncated too long message...]


   [junit4]   1>                 "core_node6":{
   [junit4]   1>                   "state":"active",
   [junit4]   1>                   "base_url":"http://127.0.0.1:50553",
   [junit4]   1>                   "core":"collection1",
   [junit4]   1>                   "node_name":"127.0.0.1:50553_"}}}}}}
   [junit4]   1>   /solr/configs (1)
   [junit4]   1>    /solr/configs/conf1 (11)
   [junit4]   1>     /solr/configs/conf1/open-exchange-rates.json (0)
   [junit4]   1>     DATA:
   [junit4]   1>         {
   [junit4]   1>           "disclaimer": "This data is not real, it was synthetically created to match currency.xml.  It is modeled after the data format available from openexchangerates.org.  See https://openexchangerates.org/documentation for details",
   [junit4]   1>           "license": "http://www.apache.org/licenses/LICENSE-2.0",
   [junit4]   1>           "timestamp": 1332070464,
   [junit4]   1>         
   [junit4]   1>         
   [junit4]   1>           "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
   [junit4]   1>         
   [junit4]   1>         
   [junit4]   1>           "base": "USD",
   [junit4]   1>           "rates": {
   [junit4]   1>             "USD": 1,
   [junit4]   1>             "JPY": 81.29,
   [junit4]   1>             "EUR": 2.5,
   [junit4]   1>             "GBP": 0.5,
   [junit4]   1>             "MXN": 2.0
   [junit4]   1>           }
   [junit4]   1>         }
   [junit4]   1>         
   [junit4]   1>     /solr/configs/conf1/solrconfig.snippet.randomindexconfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/synonyms.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/stopwords.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/currency.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/enumsConfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/protwords.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/solrconfig.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/schema.xml (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1>     /solr/configs/conf1/old_synonyms.txt (0)
   [junit4]   1>     DATA: ...supressed...
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=BE49E0FA126C0054 -Dtests.slow=true -Dtests.locale=es_UY -Dtests.timezone=Africa/Luanda -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 60.7s | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: document count mismatch.  control=70 sum(shards)=71 cloudClient=71
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BE49E0FA126C0054:3FAF6EE265336068]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1242)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:197)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 1475220 T5208 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 60744 T5207 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> 1475288 T5335 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1476400 T5353 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {other_tl1=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Memory doPackFST= true), timestamp=PostingsFormat(name=Memory doPackFST= true), text=FSTOrd41, intDefault=PostingsFormat(name=Memory doPackFST= true), a_t=PostingsFormat(name=Memory doPackFST= true), range_facet_l=PostingsFormat(name=Memory doPackFST= true), a_si=PostingsFormat(name=NestedPulsing), range_facet_si=PostingsFormat(name=NestedPulsing), range_facet_sl=FSTOrd41, id=MockVariableIntBlock(baseBlockSize=26), multiDefault=PostingsFormat(name=NestedPulsing)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=es_UY, timezone=Africa/Luanda
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=2,threads=3,free=55450216,total=222851072
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrCoreProperties, TestSolr4Spatial, TestSweetSpotSimilarityFactory, NumericFieldsTest, RAMDirectoryFactoryTest, TestHashPartitioner, TestAnalyzeInfixSuggestions, TestCollapseQParserPlugin, SearchHandlerTest, OutputWriterTest, TestRemoteStreaming, FastVectorHighlighterTest, PolyFieldTest, DefaultValueUpdateProcessorTest, TestFieldTypeResource, TestAddFieldRealTimeGet, TestFastWriter, PreAnalyzedUpdateProcessorTest, TestDocSet, PluginInfoTest, SuggesterTest, TestDistributedGrouping, SuggesterTSTTest, TestReloadAndDeleteDocs, BasicZkTest, TestRandomDVFaceting, AbstractAnalyticsStatsTest, SynonymTokenizerTest, CSVRequestHandlerTest, JsonLoaderTest, TestTrie, TestCopyFieldCollectionResource, CurrencyFieldOpenExchangeTest, UpdateRequestProcessorFactoryTest, TestPseudoReturnFields, DeleteShardTest, LukeRequestHandlerTest, TestSolrXml, TestPhraseSuggestions, CurrencyFieldXmlFileTest, TermVectorComponentDistributedTest, DebugComponentTest, SOLR749Test, DocValuesMissingTest, JSONWriterTest, ExternalFileFieldSortTest, TestUniqueKeyFieldResource, TestLazyCores, TriLevelCompositeIdRoutingTest, TestWordDelimiterFilterFactory, ClusterStateUpdateTest, RecoveryZkTest, TestLMJelinekMercerSimilarityFactory, TimeZoneUtilsTest, TestIndexSearcher, FieldAnalysisRequestHandlerTest, TestSuggestSpellingConverter, SolrCoreTest, TestNoOpRegenerator, AbstractAnalyticsFacetTest, TestMultiCoreConfBootstrap, TestFastLRUCache, FieldMutatingUpdateProcessorTest, PreAnalyzedFieldTest, TestReload, DeleteReplicaTest, TestLRUCache, FunctionTest, AlternateDirectoryTest, SolrCmdDistributorTest, TestQuerySenderNoQuery, DOMUtilTest, TestDistributedSearch, TestAtomicUpdateErrorCases, TestNonNRTOpen, TestFileDictionaryLookup, TestIBSimilarityFactory, HardAutoCommitTest, StandardRequestHandlerTest, QueryFacetTest, StatsComponentTest, SimplePostToolTest, LeaderElectionTest, TestClassNameShortening, TestSerializedLuceneMatchVersion, TestRealTimeGet, CoreAdminHandlerTest, SchemaVersionSpecificBehaviorTest, TestSolrXmlPersistence, TestStressVersions, TestDFRSimilarityFactory, TestUpdate, TestSolrXMLSerializer, TestLFUCache, TestRandomMergePolicy, TestCSVResponseWriter, TestSolrQueryParser, WordBreakSolrSpellCheckerTest, TestCloudManagedSchemaAddField, EchoParamsTest, ChangedSchemaMergeTest, DistributedSuggesterComponentTest, TestFieldTypeCollectionResource, QueryEqualityTest, TestDynamicFieldResource, SignatureUpdateProcessorFactoryTest, TestNRTOpen, TestReplicationHandler, RequestHandlersTest, HdfsBasicDistributedZk2Test, TestRTGBase, TestStressUserVersions, TestUtils, ZkNodePropsTest, TestNumberUtils, SpellingQueryConverterTest, UnloadDistributedZkTest, LeaderElectionIntegrationTest, SpellCheckCollatorTest, SuggestComponentTest, FieldFacetExtrasTest, HdfsCollectionsAPIDistributedZkTest, AssignTest, TestJmxIntegration, ZkSolrClientTest, OpenCloseCoreStressTest, TestStressLucene, AddSchemaFieldsUpdateProcessorFactoryTest, ClusterStateTest, TestWriterPerf, TestBM25SimilarityFactory, CustomCollectionTest, TestInfoStreamLogging, ResponseLogComponentTest, RegexBoostProcessorTest, TestCoreDiscovery, UpdateParamsTest, TestExtendedDismaxParser, OverseerTest, TestQueryUtils, ShowFileRequestHandlerTest, SliceStateUpdateTest, TestSystemIdResolver, SolrIndexSplitterTest, SolrTestCaseJ4Test, TestBadConfig, XmlUpdateRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestSolrDeletionPolicy1, TestCharFilters, TestSolrQueryParserDefaultOperatorResource, ZkCLITest, MultiTermTest, TestAnalyzedSuggestions, OverseerCollectionProcessorTest, SpellPossibilityIteratorTest, TermsComponentTest, TestGroupingSearch, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed in 62.04s, 1 test, 1 failure <<< FAILURES!

[...truncated 580 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:420: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:400: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1275: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:908: There were test failures: 353 suites, 1558 tests, 1 failure, 39 ignored (5 assumptions)

Total time: 82 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message