lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1141 - Failure!
Date Sun, 29 Dec 2013 16:57:25 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1141/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:53006/collection1, http://127.0.0.1:53056/collection1, http://127.0.0.1:53003/collection1, http://127.0.0.1:53031/collection1, http://127.0.0.1:53050/collection1, http://127.0.0.1:53013/collection1, http://127.0.0.1:53000/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:53006/collection1, http://127.0.0.1:53056/collection1, http://127.0.0.1:53003/collection1, http://127.0.0.1:53031/collection1, http://127.0.0.1:53050/collection1, http://127.0.0.1:53013/collection1, http://127.0.0.1:53000/collection1]
	at __randomizedtesting.SeedInfo.seed([43391F575CD2EB88:C2DF914F2B8D8BB4]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:352)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1162)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:149)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:849)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:53000/collection1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:283)
	... 46 more




Build Log:
[...truncated 10313 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 699195 T1879 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 699204 T1879 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1388333388467
   [junit4]   2> 699206 T1879 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 699207 T1880 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 699308 T1879 oasc.ZkTestServer.run start zk server on port:52993
   [junit4]   2> 699312 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 699320 T1887 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bf57f1 name:ZooKeeperConnection Watcher:127.0.0.1:52993 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 699321 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 699322 T1879 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 699336 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 699341 T1890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59c5274d name:ZooKeeperConnection Watcher:127.0.0.1:52993/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 699341 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 699342 T1879 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 699353 T1879 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 699362 T1879 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 699376 T1879 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 699390 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 699391 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 699418 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 699419 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 699450 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 699451 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 699463 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 699464 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 699477 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 699478 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 699489 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 699490 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 699502 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 699503 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 699519 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 699520 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 699530 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 699531 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 699541 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 699542 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 699549 T1879 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 699550 T1879 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 699980 T1879 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 699990 T1879 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52996
   [junit4]   2> 699991 T1879 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 699991 T1879 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 699991 T1879 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827
   [junit4]   2> 699992 T1879 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/'
   [junit4]   2> 700028 T1879 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/solr.xml
   [junit4]   2> 700157 T1879 oasc.CoreContainer.<init> New CoreContainer 1280839095
   [junit4]   2> 700158 T1879 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/]
   [junit4]   2> 700159 T1879 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 700160 T1879 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 700160 T1879 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 700160 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 700161 T1879 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 700161 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 700161 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 700162 T1879 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 700162 T1879 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 700192 T1879 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 700193 T1879 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 700193 T1879 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 700194 T1879 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52993/solr
   [junit4]   2> 700194 T1879 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 700197 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 700205 T1902 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@541185af name:ZooKeeperConnection Watcher:127.0.0.1:52993 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 700207 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 700217 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 700222 T1905 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e0ed450 name:ZooKeeperConnection Watcher:127.0.0.1:52993/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 700223 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 700228 T1879 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 700249 T1879 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 700262 T1879 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 700267 T1879 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52996_
   [junit4]   2> 700270 T1879 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52996_
   [junit4]   2> 700280 T1879 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 700297 T1879 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 700304 T1879 oasc.Overseer.start Overseer (id=90985816947228675-127.0.0.1:52996_-n_0000000000) starting
   [junit4]   2> 700316 T1879 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 700328 T1907 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 700333 T1879 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 700341 T1879 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 700347 T1879 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 700358 T1906 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 700368 T1908 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 700368 T1908 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 700371 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 700372 T1908 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 700377 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 700379 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52996",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52996_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 700379 T1906 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 700380 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 700388 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 700391 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 701374 T1908 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 701374 T1908 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/collection1
   [junit4]   2> 701375 T1908 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 701377 T1908 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 701377 T1908 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 701382 T1908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/collection1/'
   [junit4]   2> 701384 T1908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/collection1/lib/classes/' to classloader
   [junit4]   2> 701385 T1908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/collection1/lib/README' to classloader
   [junit4]   2> 701498 T1908 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 701607 T1908 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 701612 T1908 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 701625 T1908 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 702566 T1908 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 702571 T1908 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 702573 T1908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 702581 T1908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 702686 T1908 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 702715 T1908 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1388333388827/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/control/data/
   [junit4]   2> 702716 T1908 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@67a62278
   [junit4]   2> 702718 T1908 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/control/data
   [junit4]   2> 702719 T1908 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/control/data/index/
   [junit4]   2> 702720 T1908 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 702721 T1908 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/control/data/index
   [junit4]   2> 702722 T1908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2768427956270948]
   [junit4]   2> 702728 T1908 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 702730 T1908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 702734 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 702735 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 702735 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 702736 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 702736 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 702737 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 702737 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 702738 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 702738 T1908 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 702739 T1908 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 702740 T1908 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 702741 T1908 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 702741 T1908 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 702742 T1908 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 702742 T1908 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 702743 T1908 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 702754 T1908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 702758 T1908 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 702759 T1908 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 702760 T1908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 702761 T1908 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 702761 T1908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 702762 T1908 oass.SolrIndexSearcher.<init> Opening Searcher@1f9afcc6 main
   [junit4]   2> 702766 T1909 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f9afcc6 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 702768 T1908 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 702770 T1879 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 702771 T1879 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 702770 T1912 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52996 collection:control_collection shard:shard1
   [junit4]   2> 702774 T1912 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 702777 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 702783 T1915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d967118 name:ZooKeeperConnection Watcher:127.0.0.1:52993/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 702784 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 702789 T1879 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 702799 T1879 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 702802 T1912 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 702810 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 702811 T1912 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 702811 T1912 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3676 name=collection1 org.apache.solr.core.SolrCore@305528c6 url=http://127.0.0.1:52996/collection1 node=127.0.0.1:52996_ C3676_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:52996, core=collection1, node_name=127.0.0.1:52996_}
   [junit4]   2> 702811 T1912 C3676 P52996 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52996/collection1/
   [junit4]   2> 702812 T1912 C3676 P52996 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 702812 T1912 C3676 P52996 oasc.SyncStrategy.syncToMe http://127.0.0.1:52996/collection1/ has no replicas
   [junit4]   2> 702813 T1912 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52996/collection1/ shard1
   [junit4]   2> 702813 T1912 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 702821 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 702852 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 702857 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 702858 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 702885 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 702896 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 703007 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 703007 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 703035 T1912 oasc.ZkController.register We are http://127.0.0.1:52996/collection1/ and leader is http://127.0.0.1:52996/collection1/
   [junit4]   2> 703036 T1912 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52996
   [junit4]   2> 703036 T1912 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 703036 T1912 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 703052 T1912 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 703055 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 703055 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 703056 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 703058 T1912 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 703065 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 703068 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52996",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52996_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 703079 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 703190 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 703190 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 703336 T1879 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 703347 T1879 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53000
   [junit4]   2> 703349 T1879 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 703349 T1879 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 703350 T1879 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062
   [junit4]   2> 703350 T1879 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/'
   [junit4]   2> 703391 T1879 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/solr.xml
   [junit4]   2> 703551 T1879 oasc.CoreContainer.<init> New CoreContainer 1903374127
   [junit4]   2> 703552 T1879 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/]
   [junit4]   2> 703555 T1879 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 703556 T1879 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 703557 T1879 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 703557 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 703558 T1879 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 703558 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 703559 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 703560 T1879 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 703560 T1879 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 703584 T1879 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 703585 T1879 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 703585 T1879 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 703586 T1879 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52993/solr
   [junit4]   2> 703587 T1879 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 703590 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 703598 T1927 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@101c285a name:ZooKeeperConnection Watcher:127.0.0.1:52993 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 703600 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 703617 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 703625 T1930 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9704ee name:ZooKeeperConnection Watcher:127.0.0.1:52993/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 703625 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 703658 T1879 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 704671 T1879 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53000_
   [junit4]   2> 704674 T1879 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53000_
   [junit4]   2> 704685 T1930 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 704686 T1915 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 704685 T1905 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 704716 T1931 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 704717 T1931 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 704720 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 704720 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 704721 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 704720 T1931 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 704727 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 704729 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53000",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53000_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 704729 T1906 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 704729 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 704737 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 704740 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 704740 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 704741 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 705723 T1931 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 705723 T1931 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/collection1
   [junit4]   2> 705724 T1931 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 705727 T1931 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 705727 T1931 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 705731 T1931 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/collection1/'
   [junit4]   2> 705733 T1931 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/collection1/lib/classes/' to classloader
   [junit4]   2> 705734 T1931 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/collection1/lib/README' to classloader
   [junit4]   2> 705847 T1931 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 705956 T1931 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 705962 T1931 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 705974 T1931 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 706943 T1931 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 706947 T1931 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 706951 T1931 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 706962 T1931 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 707076 T1931 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 707077 T1931 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1388333392062/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty1/
   [junit4]   2> 707077 T1931 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@67a62278
   [junit4]   2> 707079 T1931 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty1
   [junit4]   2> 707079 T1931 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty1/index/
   [junit4]   2> 707080 T1931 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 707081 T1931 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty1/index
   [junit4]   2> 707082 T1931 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2768427956270948]
   [junit4]   2> 707086 T1931 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 707086 T1931 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 707093 T1931 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 707094 T1931 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 707095 T1931 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 707095 T1931 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 707096 T1931 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 707096 T1931 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 707097 T1931 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 707097 T1931 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 707098 T1931 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 707099 T1931 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 707100 T1931 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 707100 T1931 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 707100 T1931 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 707101 T1931 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 707102 T1931 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 707102 T1931 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 707116 T1931 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 707132 T1931 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 707132 T1931 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 707134 T1931 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 707136 T1931 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 707137 T1931 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 707137 T1931 oass.SolrIndexSearcher.<init> Opening Searcher@39b1601c main
   [junit4]   2> 707147 T1932 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39b1601c main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 707150 T1931 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 707152 T1879 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 707153 T1879 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 707152 T1935 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53000 collection:collection1 shard:shard2
   [junit4]   2> 707155 T1935 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 707178 T1935 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 707185 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707187 T1935 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 707187 T1935 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3677 name=collection1 org.apache.solr.core.SolrCore@222b72db url=http://127.0.0.1:53000/collection1 node=127.0.0.1:53000_ C3677_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53000, core=collection1, node_name=127.0.0.1:53000_}
   [junit4]   2> 707188 T1935 C3677 P53000 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53000/collection1/
   [junit4]   2> 707188 T1935 C3677 P53000 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 707188 T1935 C3677 P53000 oasc.SyncStrategy.syncToMe http://127.0.0.1:53000/collection1/ has no replicas
   [junit4]   2> 707189 T1935 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53000/collection1/ shard2
   [junit4]   2> 707189 T1935 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 707193 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 707206 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707209 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707209 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707212 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707237 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707243 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 707254 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707362 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707362 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707363 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707395 T1935 oasc.ZkController.register We are http://127.0.0.1:53000/collection1/ and leader is http://127.0.0.1:53000/collection1/
   [junit4]   2> 707395 T1935 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53000
   [junit4]   2> 707395 T1935 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 707396 T1935 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 707396 T1935 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 707404 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707404 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707405 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707407 T1935 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 707415 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 707417 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53000",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53000_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 707428 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 707535 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707535 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707537 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 707596 T1879 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 707632 T1879 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53003
   [junit4]   2> 707633 T1879 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 707633 T1879 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 707633 T1879 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416
   [junit4]   2> 707634 T1879 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/'
   [junit4]   2> 707680 T1879 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/solr.xml
   [junit4]   2> 707813 T1879 oasc.CoreContainer.<init> New CoreContainer 4133481
   [junit4]   2> 707814 T1879 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/]
   [junit4]   2> 707817 T1879 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 707817 T1879 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 707818 T1879 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 707818 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 707818 T1879 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 707819 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 707819 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 707819 T1879 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 707820 T1879 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 707843 T1879 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 707844 T1879 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 707844 T1879 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 707845 T1879 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52993/solr
   [junit4]   2> 707845 T1879 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 707848 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 707858 T1947 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51779b6f name:ZooKeeperConnection Watcher:127.0.0.1:52993 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 707859 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 707871 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 707875 T1950 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27395e45 name:ZooKeeperConnection Watcher:127.0.0.1:52993/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 707875 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 707890 T1879 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 708903 T1879 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53003_
   [junit4]   2> 708906 T1879 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53003_
   [junit4]   2> 708920 T1905 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 708921 T1915 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 708922 T1930 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 708922 T1950 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 708939 T1951 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 708940 T1951 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 708943 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 708944 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 708944 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 708944 T1951 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 708949 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 708952 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53003",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53003_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 708952 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 708953 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 708960 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 709069 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 709069 T1950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 709070 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 709070 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 709946 T1951 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 709946 T1951 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/collection1
   [junit4]   2> 709947 T1951 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 709949 T1951 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 709950 T1951 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 709953 T1951 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/collection1/'
   [junit4]   2> 709955 T1951 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/collection1/lib/classes/' to classloader
   [junit4]   2> 709955 T1951 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/collection1/lib/README' to classloader
   [junit4]   2> 710059 T1951 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 710171 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=523364585
   [junit4]   2> 710172 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=550086080
   [junit4]   2> 710244 T1951 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 710248 T1951 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 710269 T1951 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 711219 T1951 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 711234 T1951 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 711237 T1951 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711245 T1951 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 711335 T1951 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 711338 T1951 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1388333396416/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty2/
   [junit4]   2> 711338 T1951 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@67a62278
   [junit4]   2> 711341 T1951 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty2
   [junit4]   2> 711342 T1951 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty2/index/
   [junit4]   2> 711342 T1951 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 711344 T1951 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty2/index
   [junit4]   2> 711345 T1951 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2768427956270948]
   [junit4]   2> 711351 T1951 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 711351 T1951 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711360 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 711360 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 711361 T1951 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 711361 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711362 T1951 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 711362 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 711363 T1951 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711363 T1951 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 711363 T1951 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 711365 T1951 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 711366 T1951 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 711366 T1951 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 711368 T1951 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 711369 T1951 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 711370 T1951 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 711371 T1951 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 711382 T1951 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 711387 T1951 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 711388 T1951 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 711389 T1951 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 711392 T1951 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 711393 T1951 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 711393 T1951 oass.SolrIndexSearcher.<init> Opening Searcher@76130e87 main
   [junit4]   2> 711409 T1952 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@76130e87 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 711412 T1951 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 711414 T1879 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 711415 T1879 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 711415 T1955 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53003 collection:collection1 shard:shard1
   [junit4]   2> 711417 T1955 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 711436 T1955 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 711446 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711447 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711447 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711449 T1955 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 711449 T1955 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3678 name=collection1 org.apache.solr.core.SolrCore@72214279 url=http://127.0.0.1:53003/collection1 node=127.0.0.1:53003_ C3678_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53003, core=collection1, node_name=127.0.0.1:53003_}
   [junit4]   2> 711450 T1955 C3678 P53003 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53003/collection1/
   [junit4]   2> 711450 T1955 C3678 P53003 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 711450 T1955 C3678 P53003 oasc.SyncStrategy.syncToMe http://127.0.0.1:53003/collection1/ has no replicas
   [junit4]   2> 711451 T1955 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53003/collection1/ shard1
   [junit4]   2> 711452 T1955 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 711455 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 711473 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711479 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711481 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711481 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711482 T1950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711503 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 711514 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711621 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711621 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711622 T1950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711624 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711649 T1955 oasc.ZkController.register We are http://127.0.0.1:53003/collection1/ and leader is http://127.0.0.1:53003/collection1/
   [junit4]   2> 711650 T1955 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53003
   [junit4]   2> 711650 T1955 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 711650 T1955 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 711651 T1955 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 711654 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711655 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711656 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711659 T1955 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 711665 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 711668 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53003",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53003_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 711681 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 711789 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711790 T1950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711791 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711791 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 711869 T1879 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 711877 T1879 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53006
   [junit4]   2> 711880 T1879 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 711881 T1879 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 711881 T1879 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679
   [junit4]   2> 711882 T1879 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/'
   [junit4]   2> 711939 T1879 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/solr.xml
   [junit4]   2> 712065 T1879 oasc.CoreContainer.<init> New CoreContainer 194843638
   [junit4]   2> 712066 T1879 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/]
   [junit4]   2> 712068 T1879 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 712069 T1879 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 712069 T1879 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 712070 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 712070 T1879 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 712071 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 712071 T1879 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 712072 T1879 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 712072 T1879 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 712087 T1879 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 712088 T1879 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 712088 T1879 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 712089 T1879 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52993/solr
   [junit4]   2> 712089 T1879 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 712091 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 712096 T1967 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38060fc4 name:ZooKeeperConnection Watcher:127.0.0.1:52993 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 712098 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 712107 T1879 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 712111 T1970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5149f23d name:ZooKeeperConnection Watcher:127.0.0.1:52993/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 712112 T1879 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 712131 T1879 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 713148 T1879 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53006_
   [junit4]   2> 713151 T1879 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53006_
   [junit4]   2> 713164 T1930 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 713165 T1950 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 713165 T1905 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 713165 T1970 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 713165 T1915 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 713190 T1971 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 713190 T1971 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 713194 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 713194 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 713194 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 713194 T1971 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 713200 T1906 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 713202 T1906 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53006",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53006_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 713203 T1906 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 713203 T1906 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 713212 T1905 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 713321 T1915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 713321 T1950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 713321 T1970 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 713322 T1930 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 713321 T1905 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 714198 T1971 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 714199 T1971 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/collection1
   [junit4]   2> 714199 T1971 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 714201 T1971 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 714202 T1971 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 714206 T1971 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/collection1/'
   [junit4]   2> 714207 T1971 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/collection1/lib/classes/' to classloader
   [junit4]   2> 714208 T1971 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/collection1/lib/README' to classloader
   [junit4]   2> 714311 T1971 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 714413 T1971 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 714417 T1971 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 714437 T1971 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 715499 T1971 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 715505 T1971 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 715521 T1971 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 715530 T1971 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 715674 T1971 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 715675 T1971 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1388333400679/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty3/
   [junit4]   2> 715675 T1971 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@67a62278
   [junit4]   2> 715677 T1971 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty3
   [junit4]   2> 715678 T1971 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./o

[...truncated too long message...]

ectionContext.runLeaderProcess(ElectionContext.java:157)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:385)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:178)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:305)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:305)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1037)
   [junit4]   2> 	... 13 more
   [junit4]   2> 
   [junit4]   2> 1326727 T2279 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1326948 T2165 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1327789 T2143 C3916 P53050 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 1327790 T2143 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 1327790 T2143 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c6d74d7
   [junit4]   2> 1327792 T2143 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=23,autocommits=0,soft autocommit maxTime=1000ms,soft autocommits=22,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=156,cumulative_deletesById=73,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
   [junit4]   2> 1327793 T2143 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1327793 T2143 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1327793 T2143 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1327796 T2143 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1327809 T2143 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1327810 T2143 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty11/index [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty11/index;done=false>>]
   [junit4]   2> 1327810 T2143 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty11/index
   [junit4]   2> 1327810 T2143 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty11 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty11;done=false>>]
   [junit4]   2> 1327810 T2143 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1388333388467/jetty11
   [junit4]   2> 1327811 T2143 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1327811 T2143 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=da, timezone=Mexico/BajaSur
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=2,threads=1,free=306833816,total=519438336
   [junit4]   2> NOTE: All tests run in this JVM: [SuggestComponentTest, HdfsChaosMonkeySafeLeaderTest, TestFunctionQuery, SpellCheckCollatorTest, SpatialFilterTest, TestAddFieldRealTimeGet, TestSolrCoreProperties, TestClassNameShortening, SliceStateUpdateTest, PeerSyncTest, SliceStateTest, SolrCoreCheckLockOnStartupTest, TestLFUCache, TestCoreDiscovery, UnloadDistributedZkTest, BinaryUpdateRequestHandlerTest, DeleteInactiveReplicaTest, BasicZkTest, TestUpdate, PrimitiveFieldTypeTest, TestSolrXMLSerializer, TestRandomFaceting, TestLMJelinekMercerSimilarityFactory, CSVRequestHandlerTest, MultiTermTest, TimeZoneUtilsTest, TestPerFieldSimilarity, ShardSplitTest, AutoCommitTest, TestSchemaNameResource, DirectSolrConnectionTest, TestCustomSort, TestFileDictionaryLookup, TestSuggestSpellingConverter, HdfsSyncSliceTest, QueryEqualityTest, TestFastLRUCache, ResponseLogComponentTest, AssignTest, RegexBoostProcessorTest, MoreLikeThisHandlerTest, TestJmxIntegration, SolrRequestParserTest, DocValuesMultiTest, EnumFieldTest, AddBlockUpdateTest, TestCollationField, DistributedDebugComponentTest, ZkNodePropsTest, DistributedTermsComponentTest, TestSchemaResource, IndexBasedSpellCheckerTest, TestManagedSchemaFieldResource, OverseerCollectionProcessorTest, TestSolr4Spatial, StandardRequestHandlerTest, TestRTGBase, TestQueryTypes, TestSolrXmlPersistor, HdfsBasicDistributedZkTest, TestCoreContainer, BlockDirectoryTest, FileBasedSpellCheckerTest, DistanceFunctionTest, FullSolrCloudDistribCmdsTest, SolrPluginUtilsTest, NumericFieldsTest, ZkCLITest, TestFieldTypeResource, TestSearchPerf, SuggesterWFSTTest, StatsComponentTest, SignatureUpdateProcessorFactoryTest, TestCharFilters, CurrencyFieldXmlFileTest, TestPhraseSuggestions, RAMDirectoryFactoryTest, TestValueSourceCache, TestRecoveryHdfs, ClusterStateUpdateTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed in 629.40s, 1 test, 1 error <<< FAILURES!

[...truncated 816 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1307: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:940: There were test failures: 348 suites, 1519 tests, 1 error, 40 ignored (5 assumptions)

Total time: 93 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message