lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1080 - Still Failing!
Date Sun, 01 Dec 2013 22:59:00 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1080/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:50592/collection1, http://127.0.0.1:50598/collection1, http://127.0.0.1:50595/collection1, http://127.0.0.1:50644/collection1, http://127.0.0.1:50605/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:50592/collection1, http://127.0.0.1:50598/collection1, http://127.0.0.1:50595/collection1, http://127.0.0.1:50644/collection1, http://127.0.0.1:50605/collection1]
	at __randomizedtesting.SeedInfo.seed([4BC629E9AD063D50:CA20A7F1DA595D6C]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:354)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1162)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:147)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:843)
	at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:50592/collection1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:491)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:285)
	... 45 more




Build Log:
[...truncated 10406 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 1681369 T4652 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1681382 T4652 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1385936984536
   [junit4]   2> 1681384 T4652 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1681385 T4653 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1681486 T4652 oasc.ZkTestServer.run start zk server on port:50585
   [junit4]   2> 1681488 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1681494 T4659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a92874b name:ZooKeeperConnection Watcher:127.0.0.1:50585 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1681494 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1681495 T4652 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1681504 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1681508 T4661 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a903201 name:ZooKeeperConnection Watcher:127.0.0.1:50585/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1681508 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1681509 T4652 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1681515 T4652 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1681521 T4652 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1681527 T4652 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1681532 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1681533 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1681543 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1681543 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1681550 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1681550 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1681557 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1681557 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1681563 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1681563 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1681569 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1681570 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1681576 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1681576 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1681581 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1681581 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1681587 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1681587 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1681595 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1681596 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1681602 T4652 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1681603 T4652 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1681921 T4652 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1681927 T4652 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50588
   [junit4]   2> 1681927 T4652 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1681928 T4652 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1681928 T4652 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763
   [junit4]   2> 1681928 T4652 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/'
   [junit4]   2> 1681957 T4652 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/solr.xml
   [junit4]   2> 1682022 T4652 oasc.CoreContainer.<init> New CoreContainer 268636776
   [junit4]   2> 1682023 T4652 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/]
   [junit4]   2> 1682024 T4652 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1682024 T4652 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1682025 T4652 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1682025 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1682025 T4652 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1682025 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1682026 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1682026 T4652 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1682026 T4652 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1682039 T4652 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1682040 T4652 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1682040 T4652 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1682040 T4652 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50585/solr
   [junit4]   2> 1682041 T4652 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1682042 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1682047 T4672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2278ab9 name:ZooKeeperConnection Watcher:127.0.0.1:50585 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1682048 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1682052 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1682055 T4674 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27e2ee7c name:ZooKeeperConnection Watcher:127.0.0.1:50585/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1682056 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1682059 T4652 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1682066 T4652 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1682072 T4652 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1682076 T4652 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50588_
   [junit4]   2> 1682078 T4652 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50588_
   [junit4]   2> 1682084 T4652 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1682095 T4652 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1682099 T4652 oasc.Overseer.start Overseer (id=90828766219010051-127.0.0.1:50588_-n_0000000000) starting
   [junit4]   2> 1682107 T4652 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1682118 T4676 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1682119 T4652 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1682125 T4652 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1682129 T4652 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1682135 T4675 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1682144 T4677 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1682144 T4677 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1682146 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1682146 T4677 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1682151 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1682152 T4675 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50588",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50588_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1682152 T4675 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1682153 T4675 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1682159 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1682160 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1683148 T4677 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1683149 T4677 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/collection1
   [junit4]   2> 1683149 T4677 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1683151 T4677 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1683151 T4677 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1683153 T4677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/collection1/'
   [junit4]   2> 1683155 T4677 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/collection1/lib/classes/' to classloader
   [junit4]   2> 1683155 T4677 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/collection1/lib/README' to classloader
   [junit4]   2> 1683217 T4677 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1683305 T4677 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1683309 T4677 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1683321 T4677 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1683995 T4677 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1684001 T4677 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1684003 T4677 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1684011 T4677 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1684085 T4677 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1684085 T4677 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1385936984763/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/control/data/
   [junit4]   2> 1684085 T4677 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14121689
   [junit4]   2> 1684087 T4677 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/control/data
   [junit4]   2> 1684089 T4677 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/control/data/index/
   [junit4]   2> 1684089 T4677 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1684090 T4677 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/control/data/index
   [junit4]   2> 1684091 T4677 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1831885867, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1684097 T4677 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1684098 T4677 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1684105 T4677 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1684106 T4677 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1684106 T4677 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1684106 T4677 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1684106 T4677 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1684107 T4677 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1684107 T4677 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1684107 T4677 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1684108 T4677 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1684109 T4677 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1684110 T4677 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1684110 T4677 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1684110 T4677 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1684111 T4677 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1684111 T4677 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1684112 T4677 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1684122 T4677 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1684126 T4677 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1684127 T4677 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1684127 T4677 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=67.951171875, floorSegmentMB=1.0302734375, forceMergeDeletesPctAllowed=14.426797007620774, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1684129 T4677 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1684129 T4677 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1684130 T4677 oass.SolrIndexSearcher.<init> Opening Searcher@9d8bf55 main
   [junit4]   2> 1684133 T4678 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9d8bf55 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1684134 T4677 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1684134 T4677 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50588 collection:control_collection shard:shard1
   [junit4]   2> 1684136 T4677 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1684162 T4677 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1684168 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1684171 T4677 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1684172 T4677 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1254 name=collection1 org.apache.solr.core.SolrCore@58b3f9c5 url=http://127.0.0.1:50588/collection1 node=127.0.0.1:50588_ C1254_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:50588, core=collection1, node_name=127.0.0.1:50588_}
   [junit4]   2> 1684172 T4677 C1254 P50588 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50588/collection1/
   [junit4]   2> 1684172 T4677 C1254 P50588 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1684173 T4677 C1254 P50588 oasc.SyncStrategy.syncToMe http://127.0.0.1:50588/collection1/ has no replicas
   [junit4]   2> 1684173 T4677 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50588/collection1/ shard1
   [junit4]   2> 1684174 T4677 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1684175 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1684188 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1684192 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1684207 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1684216 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1684322 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1684356 T4677 oasc.ZkController.register We are http://127.0.0.1:50588/collection1/ and leader is http://127.0.0.1:50588/collection1/
   [junit4]   2> 1684357 T4677 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50588
   [junit4]   2> 1684357 T4677 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1684358 T4677 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1684358 T4677 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1684361 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1684361 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1684362 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1684362 T4677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1684365 T4652 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1684366 T4652 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1684366 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1684368 T4675 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50588",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50588_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1684369 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1684374 T4681 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@537c82f1 name:ZooKeeperConnection Watcher:127.0.0.1:50585/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1684375 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1684376 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1684378 T4652 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1684385 T4652 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1684482 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1684482 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1684705 T4652 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1684708 T4652 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50592
   [junit4]   2> 1684711 T4652 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1684711 T4652 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1684711 T4652 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538
   [junit4]   2> 1684711 T4652 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/'
   [junit4]   2> 1684744 T4652 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/solr.xml
   [junit4]   2> 1684812 T4652 oasc.CoreContainer.<init> New CoreContainer 1528700438
   [junit4]   2> 1684812 T4652 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/]
   [junit4]   2> 1684814 T4652 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1684814 T4652 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1684814 T4652 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1684814 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1684815 T4652 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1684815 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1684815 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1684815 T4652 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1684815 T4652 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1684830 T4652 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1684831 T4652 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1684831 T4652 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1684831 T4652 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50585/solr
   [junit4]   2> 1684832 T4652 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1684833 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1684837 T4692 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@178e6551 name:ZooKeeperConnection Watcher:127.0.0.1:50585 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1684838 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1684843 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1684848 T4694 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40c1d04c name:ZooKeeperConnection Watcher:127.0.0.1:50585/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1684849 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1684858 T4652 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1685871 T4652 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50592_
   [junit4]   2> 1685874 T4652 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50592_
   [junit4]   2> 1685881 T4694 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1685882 T4681 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1685882 T4674 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1685896 T4695 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1685896 T4695 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1685899 T4695 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1685899 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1685900 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1685900 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1685905 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1685906 T4675 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50592",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50592_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1685907 T4675 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1685907 T4675 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1685914 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1686023 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1686023 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1686023 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1686904 T4695 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1686904 T4695 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/collection1
   [junit4]   2> 1686904 T4695 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1686906 T4695 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1686906 T4695 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1686909 T4695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/collection1/'
   [junit4]   2> 1686911 T4695 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/collection1/lib/classes/' to classloader
   [junit4]   2> 1686911 T4695 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/collection1/lib/README' to classloader
   [junit4]   2> 1686973 T4695 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1687029 T4695 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1687033 T4695 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1687046 T4695 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1687547 T4695 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1687551 T4695 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1687553 T4695 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1687560 T4695 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1687616 T4695 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1687616 T4695 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1385936987538/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty1/
   [junit4]   2> 1687616 T4695 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14121689
   [junit4]   2> 1687617 T4695 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty1
   [junit4]   2> 1687618 T4695 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty1/index/
   [junit4]   2> 1687618 T4695 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1687620 T4695 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty1/index
   [junit4]   2> 1687621 T4695 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1831885867, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1687625 T4695 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1687626 T4695 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1687640 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1687641 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1687641 T4695 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1687641 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1687641 T4695 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1687642 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1687642 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1687642 T4695 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1687642 T4695 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1687643 T4695 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1687644 T4695 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1687644 T4695 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1687645 T4695 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1687645 T4695 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1687645 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1687646 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1687656 T4695 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1687660 T4695 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1687660 T4695 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1687661 T4695 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=67.951171875, floorSegmentMB=1.0302734375, forceMergeDeletesPctAllowed=14.426797007620774, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1687663 T4695 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1687663 T4695 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1687663 T4695 oass.SolrIndexSearcher.<init> Opening Searcher@6b58cbbf main
   [junit4]   2> 1687670 T4696 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b58cbbf main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1687673 T4695 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1687673 T4695 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50592 collection:collection1 shard:shard1
   [junit4]   2> 1687675 T4695 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1687694 T4695 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1687699 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687699 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687700 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687700 T4695 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1687700 T4695 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1255 name=collection1 org.apache.solr.core.SolrCore@36b8c17e url=http://127.0.0.1:50592/collection1 node=127.0.0.1:50592_ C1255_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50592, core=collection1, node_name=127.0.0.1:50592_}
   [junit4]   2> 1687701 T4695 C1255 P50592 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50592/collection1/
   [junit4]   2> 1687701 T4695 C1255 P50592 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1687701 T4695 C1255 P50592 oasc.SyncStrategy.syncToMe http://127.0.0.1:50592/collection1/ has no replicas
   [junit4]   2> 1687701 T4695 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50592/collection1/ shard1
   [junit4]   2> 1687702 T4695 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1687703 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1687710 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687713 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1687713 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1687713 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1687723 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1687731 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687837 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1687837 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1687838 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1687879 T4695 oasc.ZkController.register We are http://127.0.0.1:50592/collection1/ and leader is http://127.0.0.1:50592/collection1/
   [junit4]   2> 1687879 T4695 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50592
   [junit4]   2> 1687879 T4695 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1687879 T4695 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1687879 T4695 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1687882 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687882 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687882 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1687883 T4695 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1687885 T4652 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1687886 T4652 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1687885 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1687887 T4675 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50592",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50592_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1687899 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1688002 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1688003 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1688003 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1688201 T4652 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1688205 T4652 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50595
   [junit4]   2> 1688207 T4652 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1688208 T4652 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1688208 T4652 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039
   [junit4]   2> 1688208 T4652 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/'
   [junit4]   2> 1688239 T4652 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/solr.xml
   [junit4]   2> 1688306 T4652 oasc.CoreContainer.<init> New CoreContainer 876467191
   [junit4]   2> 1688307 T4652 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/]
   [junit4]   2> 1688313 T4652 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1688313 T4652 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1688314 T4652 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1688314 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1688314 T4652 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1688314 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1688314 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1688315 T4652 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1688315 T4652 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1688335 T4652 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1688335 T4652 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1688335 T4652 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1688336 T4652 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50585/solr
   [junit4]   2> 1688336 T4652 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1688337 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1688342 T4708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@407579a name:ZooKeeperConnection Watcher:127.0.0.1:50585 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1688343 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1688348 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1688353 T4710 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4cebc996 name:ZooKeeperConnection Watcher:127.0.0.1:50585/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1688353 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1688364 T4652 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1689373 T4652 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50595_
   [junit4]   2> 1689376 T4652 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50595_
   [junit4]   2> 1689383 T4694 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1689384 T4674 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1689383 T4710 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1689384 T4681 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1689397 T4711 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1689397 T4711 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1689399 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1689400 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1689400 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1689399 T4711 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1689404 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1689406 T4675 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50595",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50595_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1689406 T4675 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1689406 T4675 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1689413 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1689519 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1689519 T4710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1689519 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1689519 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1690402 T4711 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1690402 T4711 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/collection1
   [junit4]   2> 1690403 T4711 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1690405 T4711 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1690405 T4711 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1690409 T4711 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/collection1/'
   [junit4]   2> 1690410 T4711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/collection1/lib/classes/' to classloader
   [junit4]   2> 1690411 T4711 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/collection1/lib/README' to classloader
   [junit4]   2> 1690470 T4711 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1690526 T4711 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1690530 T4711 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1690537 T4711 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1691106 T4711 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1691108 T4711 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1691110 T4711 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1691117 T4711 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1691174 T4711 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1691174 T4711 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1385936991039/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty2/
   [junit4]   2> 1691175 T4711 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14121689
   [junit4]   2> 1691175 T4711 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty2
   [junit4]   2> 1691176 T4711 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty2/index/
   [junit4]   2> 1691177 T4711 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1691178 T4711 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty2/index
   [junit4]   2> 1691179 T4711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1831885867, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1691182 T4711 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1691182 T4711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1691187 T4711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1691187 T4711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1691188 T4711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1691188 T4711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1691188 T4711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1691188 T4711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1691189 T4711 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1691189 T4711 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1691189 T4711 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1691190 T4711 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1691191 T4711 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1691191 T4711 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1691192 T4711 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1691192 T4711 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1691193 T4711 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1691193 T4711 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1691203 T4711 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1691207 T4711 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1691207 T4711 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1691208 T4711 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=67.951171875, floorSegmentMB=1.0302734375, forceMergeDeletesPctAllowed=14.426797007620774, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1691209 T4711 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1691209 T4711 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1691210 T4711 oass.SolrIndexSearcher.<init> Opening Searcher@48867d81 main
   [junit4]   2> 1691216 T4712 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48867d81 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1691217 T4711 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1691217 T4711 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50595 collection:collection1 shard:shard3
   [junit4]   2> 1691219 T4711 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 1691235 T4711 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 1691241 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691242 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691242 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691242 T4711 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1691242 T4711 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1256 name=collection1 org.apache.solr.core.SolrCore@7abc8d77 url=http://127.0.0.1:50595/collection1 node=127.0.0.1:50595_ C1256_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50595, core=collection1, node_name=127.0.0.1:50595_}
   [junit4]   2> 1691243 T4711 C1256 P50595 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50595/collection1/
   [junit4]   2> 1691243 T4711 C1256 P50595 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1691243 T4711 C1256 P50595 oasc.SyncStrategy.syncToMe http://127.0.0.1:50595/collection1/ has no replicas
   [junit4]   2> 1691243 T4711 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50595/collection1/ shard3
   [junit4]   2> 1691244 T4711 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 1691245 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1691251 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691253 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691253 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691254 T4710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691253 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691270 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1691279 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691386 T4710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691386 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691386 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691386 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691416 T4711 oasc.ZkController.register We are http://127.0.0.1:50595/collection1/ and leader is http://127.0.0.1:50595/collection1/
   [junit4]   2> 1691416 T4711 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50595
   [junit4]   2> 1691417 T4711 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1691417 T4711 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1691417 T4711 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1691420 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691421 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691421 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691422 T4711 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1691424 T4652 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1691425 T4652 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1691424 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1691427 T4675 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50595",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50595_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1691442 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1691546 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691547 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691548 T4710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691549 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1691746 T4652 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1691749 T4652 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50598
   [junit4]   2> 1691751 T4652 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1691752 T4652 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1691752 T4652 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577
   [junit4]   2> 1691752 T4652 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/'
   [junit4]   2> 1691782 T4652 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/solr.xml
   [junit4]   2> 1691849 T4652 oasc.CoreContainer.<init> New CoreContainer 1841003247
   [junit4]   2> 1691849 T4652 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/]
   [junit4]   2> 1691850 T4652 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1691851 T4652 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1691851 T4652 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1691851 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1691851 T4652 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1691852 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1691853 T4652 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1691853 T4652 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1691853 T4652 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1691866 T4652 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1691867 T4652 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1691867 T4652 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1691868 T4652 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50585/solr
   [junit4]   2> 1691868 T4652 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1691869 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1691874 T4724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c4a7ec3 name:ZooKeeperConnection Watcher:127.0.0.1:50585 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1691876 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1691882 T4652 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1691887 T4726 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e4c2a85 name:ZooKeeperConnection Watcher:127.0.0.1:50585/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1691887 T4652 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1691898 T4652 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1692907 T4652 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50598_
   [junit4]   2> 1692910 T4652 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50598_
   [junit4]   2> 1692917 T4674 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1692918 T4726 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1692918 T4710 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1692918 T4694 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1692918 T4681 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1692934 T4727 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1692934 T4727 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1692937 T4727 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1692937 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1692937 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1692938 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1692941 T4675 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1692943 T4675 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50598",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50598_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1692943 T4675 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1692943 T4675 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1692950 T4674 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1693055 T4674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1693056 T4726 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1693056 T4681 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1693056 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1693056 T4710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1693939 T4727 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1693939 T4727 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/collection1
   [junit4]   2> 1693939 T4727 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1693941 T4727 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1693941 T4727 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1693944 T4727 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/collection1/'
   [junit4]   2> 1693945 T4727 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/collection1/lib/classes/' to classloader
   [junit4]   2> 1693946 T4727 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/collection1/lib/README' to classloader
   [junit4]   2> 1694017 T4727 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1694067 T4727 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1694071 T4727 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1694079 T4727 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1694579 T4727 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1694582 T4727 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1694584 T4727 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1694591 T4727 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1694650 T4727 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1694651 T4727 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1385936994577/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty3/
   [junit4]   2> 1694651 T4727 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14121689
   [junit4]   2> 1694652 T4727 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty3
   [junit4]   2> 1694653 T4727 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty3/index/
   [junit4]   2> 1694654 T4727 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 1694655 T4727 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty3/index
   [junit4]   2> 1694656 T4727 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1831885867, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1694659 T4727 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 1694660 T4727 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1694663 T4727 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1694664 T4727 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1694664 T4727 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProces

[...truncated too long message...]

lr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty12/index.20131202033031902 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty12/index.20131202033031902;done=false>>]
   [junit4]   2> 1872626 T4652 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty12/index.20131202033031902
   [junit4]   2> 1872626 T4652 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty12 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty12;done=false>>]
   [junit4]   2> 1872627 T4652 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1385936984535/jetty12
   [junit4]   2> 1872627 T4885 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1872631 T4885 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1872644 T4652 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1873035 T4867 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1874029 T4652 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1874033 T4652 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50585 50585
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   1> safe stop:
   [junit4]   1> safe stop:
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=4BC629E9AD063D50 -Dtests.slow=true -Dtests.locale=sr_ME -Dtests.timezone=Indian/Kerguelen -Dtests.file.encoding=UTF-8
   [junit4] ERROR    193s | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:50592/collection1, http://127.0.0.1:50598/collection1, http://127.0.0.1:50595/collection1, http://127.0.0.1:50644/collection1, http://127.0.0.1:50605/collection1]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4BC629E9AD063D50:CA20A7F1DA595D6C]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:354)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:640)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1162)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:147)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:843)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:50592/collection1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:491)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:285)
   [junit4]    > 	... 45 more
   [junit4]   2> 1874143 T4652 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 192779 T4651 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=sr_ME, timezone=Indian/Kerguelen
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=2,threads=1,free=206509880,total=353894400
   [junit4]   2> NOTE: All tests run in this JVM: [ShowFileRequestHandlerTest, TestSchemaResource, JsonLoaderTest, TestQuerySenderListener, TestStressLucene, TestFoldingMultitermQuery, CurrencyFieldXmlFileTest, UpdateParamsTest, TestReload, TestCoreContainer, URLClassifyProcessorTest, BadIndexSchemaTest, UUIDFieldTest, TestSolrXml, BasicZkTest, TestRandomFaceting, CoreContainerCoreInitFailuresTest, TestIndexingPerformance, FieldFacetExtrasTest, JSONWriterTest, DeleteShardTest, EchoParamsTest, XsltUpdateRequestHandlerTest, DocValuesMultiTest, TestElisionMultitermQuery, ZkNodePropsTest, SearchHandlerTest, TestStressVersions, PathHierarchyTokenizerFactoryTest, PrimUtilsTest, TestValueSourceCache, AbstractAnalyticsFacetTest, CustomCollectionTest, SpatialFilterTest, DateMathParserTest, NotRequiredUniqueKeyTest, TestDocBasedVersionConstraints, PreAnalyzedUpdateProcessorTest, CoreMergeIndexesAdminHandlerTest, TestDistributedSearch, LeaderElectionTest, CoreAdminHandlerTest, TestOmitPositions, ScriptEngineTest, StressHdfsTest, DistributedSuggesterComponentTest, TestFastLRUCache, AddSchemaFieldsUpdateProcessorFactoryTest, TestCSVLoader, TestJoin, InfoHandlerTest, TestSchemaSimilarityResource, TermVectorComponentDistributedTest, DistributedSpellCheckComponentTest, LeaderElectionIntegrationTest, TestCloudManagedSchemaAddField, TestFieldResource, RecoveryZkTest, BadCopyFieldTest, ParsingFieldUpdateProcessorsTest, TestSchemaNameResource, MinimalSchemaTest, CollectionsAPIDistributedZkTest, FileBasedSpellCheckerTest, AnalysisAfterCoreReloadTest, TestDocumentBuilder, BinaryUpdateRequestHandlerTest, TestSolrQueryParserDefaultOperatorResource, PolyFieldTest, FullSolrCloudDistribCmdsTest, TestCustomSort, SimplePostToolTest, SuggesterTest, HdfsSyncSliceTest, TestRTGBase, TermVectorComponentTest, TestSolrXMLSerializer, TestFastWriter, TestFieldTypeResource, TestComponentsName, DocumentBuilderTest, SuggesterTSTTest, TestConfig, TestStressUserVersions, SynonymTokenizerTest, TestUtils, TestUniqueKeyFieldResource, ConvertedLegacyTest, TestSolrQueryParser, FileUtilsTest, TestJmxIntegration, TestSuggestSpellingConverter, OpenCloseCoreStressTest, TestFieldTypeCollectionResource, TestManagedSchemaFieldResource, DistributedQueryElevationComponentTest, XmlUpdateRequestHandlerTest, MigrateRouteKeyTest, QueryParsingTest, TestLRUCache, MoreLikeThisHandlerTest, OverseerTest, TimeZoneUtilsTest, SignatureUpdateProcessorFactoryTest, TestCollationField, TestSystemIdResolver, TestCollationFieldDocValues, TestUpdate, QueryEqualityTest, HdfsBasicDistributedZk2Test, LoggingHandlerTest, TestTrie, TestRealTimeGet, TestShardHandlerFactory, ClusterStateTest, SolrIndexSplitterTest, DeleteReplicaTest, TestFaceting, TestFunctionQuery, TestQueryTypes, ChangedSchemaMergeTest, ZkControllerTest, ResourceLoaderTest, RequestHandlersTest, LegacyHTMLStripCharFilterTest, TestSolrIndexConfig, TestLMJelinekMercerSimilarityFactory, TestCopyFieldCollectionResource, CopyFieldTest, PingRequestHandlerTest, AlternateDirectoryTest, DocValuesTest, DOMUtilTest, TestSearchPerf, TestNoOpRegenerator, TestAddFieldRealTimeGet, TestDefaultSearchFieldResource, TestDFRSimilarityFactory, IndexSchemaTest, SortByFunctionTest, TestAtomicUpdateErrorCases, TestCodecSupport, TestRandomMergePolicy, TestSweetSpotSimilarityFactory, NoFacetTest, BasicFunctionalityTest, SliceStateUpdateTest, TestHighFrequencyDictionaryFactory, DirectUpdateHandlerOptimizeTest, ExternalFileFieldSortTest, TestDistribDocBasedVersion, HdfsBasicDistributedZkTest, SystemInfoHandlerTest, DistributedTermsComponentTest, SolrXmlInZkTest, SOLR749Test, AutoCommitTest, ResponseLogComponentTest, StatsComponentTest, DateFieldTest, OutputWriterTest, TestPartialUpdateDeduplication, TestBinaryField, TestFastOutputStream, QueryElevationComponentTest, PrimitiveFieldTypeTest, TestRemoteStreaming, SpellCheckComponentTest, TestArbitraryIndexDir, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed in 193.57s, 1 test, 1 error <<< FAILURES!

[...truncated 548 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:420: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:400: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1275: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:908: There were test failures: 357 suites, 1570 tests, 1 error, 39 ignored (5 assumptions)

Total time: 86 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message