lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1294 - Failure!
Date Thu, 13 Feb 2014 23:09:37 GMT

Stange - seems to be stuck on:

   [junit4]   2> 	  17) Thread[id=2057, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[7942C2603C4247B0], state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.security.AccessController.getStackAccessControlContext(Native Method)
   [junit4]   2> 	        at java.security.AccessController.checkPermission(AccessController.java:533)
   [junit4]   2> 	        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
   [junit4]   2> 	        at java.lang.SecurityManager.checkDelete(SecurityManager.java:1007)
   [junit4]   2> 	        at java.io.File.delete(File.java:1030)
   [junit4]   2> 	        at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:272)
   [junit4]   2> 	        at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:584)
   [junit4]   2> 	        at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517)
   [junit4]   2> 	        at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:286)
   [junit4]   2> 	        at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:457)
   [junit4]   2> 	        at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3049)
   [junit4]   2> 	        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3029)
   [junit4]   2> 	        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
   [junit4]   2> 	        at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:765)
   [junit4]   2> 	        at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:68)
   [junit4]   2> 	        at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:359)
   [junit4]   2> 	        at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:72)
   [junit4]   2> 	        at org.apache.solr.core.SolrCore.close(SolrCore.java:1043)
   [junit4]   2> 	        at org.apache.solr.core.SolrCores.close(SolrCores.java:120)
   [junit4]   2> 	        at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:386)
   [junit4]   2> 	        at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:210)
   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:224)
   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:563)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1571)
   [junit4]   2> 	        at org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:288)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:222)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1552)
   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tearDown(ChaosMonkeySafeLeaderTest.java:84)


- Mark

http://about.me/markrmiller

On Feb 13, 2014, at 5:41 PM, Policeman Jenkins Server <jenkins@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1294/
> Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
> 
> 2 tests failed.
> REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
> 
> Error Message:
> Test abandoned because suite timeout was reached.
> 
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
> 	at __randomizedtesting.SeedInfo.seed([7942C2603C4247B0]:0)
> 
> 
> FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
> 
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
> 
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
> 	at __randomizedtesting.SeedInfo.seed([7942C2603C4247B0]:0)
> 
> 
> 
> 
> Build Log:
> [...truncated 19444 lines...]
>   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
>   [junit4]   2> 452322 T2057 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ul/ki
>   [junit4]   2> 452343 T2057 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
>   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1392323938383
>   [junit4]   2> 452345 T2057 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>   [junit4]   2> 452346 T2058 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>   [junit4]   2> 452460 T2057 oasc.ZkTestServer.run start zk server on port:59284
>   [junit4]   2> 452467 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 452498 T2064 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e7764da name:ZooKeeperConnection Watcher:127.0.0.1:59284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 452504 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 452505 T2057 oascc.SolrZkClient.makePath makePath: /solr
>   [junit4]   2> 452553 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 452559 T2066 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33742169 name:ZooKeeperConnection Watcher:127.0.0.1:59284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 452560 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 452560 T2057 oascc.SolrZkClient.makePath makePath: /collections/collection1
>   [junit4]   2> 452575 T2057 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
>   [junit4]   2> 452589 T2057 oascc.SolrZkClient.makePath makePath: /collections/control_collection
>   [junit4]   2> 452596 T2057 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
>   [junit4]   2> 452603 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
>   [junit4]   2> 452603 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
>   [junit4]   2> 452615 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
>   [junit4]   2> 452616 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
>   [junit4]   2> 452623 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 452624 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>   [junit4]   2> 452632 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
>   [junit4]   2> 452632 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
>   [junit4]   2> 452640 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
>   [junit4]   2> 452640 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
>   [junit4]   2> 452647 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
>   [junit4]   2> 452647 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
>   [junit4]   2> 452656 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
>   [junit4]   2> 452656 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
>   [junit4]   2> 452664 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 452665 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
>   [junit4]   2> 452673 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 452674 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
>   [junit4]   2> 452682 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
>   [junit4]   2> 452683 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
>   [junit4]   2> 452691 T2057 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
>   [junit4]   2> 452691 T2057 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
>   [junit4]   2> 453037 T2057 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 453045 T2057 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59287
>   [junit4]   2> 453046 T2057 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 453047 T2057 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 453047 T2057 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740
>   [junit4]   2> 453047 T2057 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/'
>   [junit4]   2> 453078 T2057 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/solr.xml
>   [junit4]   2> 453169 T2057 oasc.CoreContainer.<init> New CoreContainer 657112940
>   [junit4]   2> 453169 T2057 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/]
>   [junit4]   2> 453171 T2057 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 453171 T2057 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 453171 T2057 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 453171 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 453171 T2057 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 453172 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 453172 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 453172 T2057 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 453172 T2057 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 453183 T2057 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 453184 T2057 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 453184 T2057 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 453184 T2057 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59284/solr
>   [junit4]   2> 453184 T2057 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 453186 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 453191 T2077 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@677d5f8a name:ZooKeeperConnection Watcher:127.0.0.1:59284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 453191 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 453198 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 453202 T2079 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ab6ed6f name:ZooKeeperConnection Watcher:127.0.0.1:59284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 453203 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 453207 T2057 oascc.SolrZkClient.makePath makePath: /overseer/queue
>   [junit4]   2> 453216 T2057 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
>   [junit4]   2> 453226 T2057 oascc.SolrZkClient.makePath makePath: /live_nodes
>   [junit4]   2> 453230 T2057 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59287_ul%2Fki
>   [junit4]   2> 453232 T2057 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59287_ul%2Fki
>   [junit4]   2> 453242 T2057 oascc.SolrZkClient.makePath makePath: /overseer_elect
>   [junit4]   2> 453248 T2057 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
>   [junit4]   2> 453257 T2057 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:59287_ul%2Fki
>   [junit4]   2> 453257 T2057 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
>   [junit4]   2> 453264 T2057 oasc.Overseer.start Overseer (id=91247341626392579-127.0.0.1:59287_ul%2Fki-n_0000000000) starting
>   [junit4]   2> 453272 T2057 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
>   [junit4]   2> 453284 T2081 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
>   [junit4]   2> 453286 T2057 oascc.SolrZkClient.makePath makePath: /clusterstate.json
>   [junit4]   2> 453287 T2081 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
>   [junit4]   2> 453292 T2057 oascc.SolrZkClient.makePath makePath: /aliases.json
>   [junit4]   2> 453296 T2057 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 453305 T2080 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
>   [junit4]   2> 453312 T2082 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 453312 T2082 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 453315 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 453315 T2082 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 453320 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 453322 T2080 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:59287/ul/ki",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:59287_ul%2Fki",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"control_collection",
>   [junit4]   2> 	  "numShards":"1",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 453322 T2080 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
>   [junit4]   2> 453322 T2080 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
>   [junit4]   2> 453329 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 453331 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 454316 T2082 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 454316 T2082 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/collection1
>   [junit4]   2> 454316 T2082 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
>   [junit4]   2> 454318 T2082 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 454319 T2082 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
>   [junit4]   2> 454322 T2082 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
>   [junit4]   2> 454322 T2082 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/collection1/'
>   [junit4]   2> 454323 T2082 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/collection1/lib/classes/' to classloader
>   [junit4]   2> 454324 T2082 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/collection1/lib/README' to classloader
>   [junit4]   2> 454389 T2082 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 454457 T2082 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 454460 T2082 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 454469 T2082 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 455129 T2082 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 455134 T2082 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 455137 T2082 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 455143 T2082 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 455218 T2082 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>   [junit4]   2> 455219 T2082 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392323938740/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/control/data/
>   [junit4]   2> 455219 T2082 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b3d4e26
>   [junit4]   2> 455220 T2082 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/control/data
>   [junit4]   2> 455221 T2082 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/control/data/index/
>   [junit4]   2> 455221 T2082 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/control/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 455222 T2082 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/control/data/index
>   [junit4]   2> 455222 T2082 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5503282300917641]
>   [junit4]   2> 455227 T2082 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/control/data/index,segFN=segments_1,generation=1}
>   [junit4]   2> 455227 T2082 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 455233 T2082 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 455234 T2082 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 455235 T2082 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 455235 T2082 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 455236 T2082 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 455236 T2082 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 455236 T2082 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 455237 T2082 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 455237 T2082 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 455237 T2082 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 455238 T2082 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 455238 T2082 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 455238 T2082 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 455239 T2082 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 455239 T2082 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 455239 T2082 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 455249 T2082 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 455254 T2082 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 455254 T2082 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 455255 T2082 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=36.1337890625, floorSegmentMB=1.6708984375, forceMergeDeletesPctAllowed=6.1288322191713505, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>   [junit4]   2> 455257 T2082 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/control/data/index,segFN=segments_1,generation=1}
>   [junit4]   2> 455257 T2082 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 455257 T2082 oass.SolrIndexSearcher.<init> Opening Searcher@32bd453[collection1] main
>   [junit4]   2> 455261 T2083 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@32bd453[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 455262 T2082 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 455264 T2057 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 455264 T2086 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59287/ul/ki collection:control_collection shard:shard1
>   [junit4]   2> 455264 T2057 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 455266 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 455268 T2086 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
>   [junit4]   2> 455274 T2088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b70afe0 name:ZooKeeperConnection Watcher:127.0.0.1:59284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 455274 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 455276 T2057 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 455281 T2057 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
>   [junit4]   2> 455289 T2086 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 455293 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 455295 T2086 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 455295 T2086 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C687 name=collection1 org.apache.solr.core.SolrCore@4d93e103 url=http://127.0.0.1:59287/ul/ki/collection1 node=127.0.0.1:59287_ul%2Fki C687_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:59287/ul/ki, core=collection1, node_name=127.0.0.1:59287_ul%2Fki}
>   [junit4]   2> 455295 T2086 C687 P59287 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59287/ul/ki/collection1/
>   [junit4]   2> 455295 T2086 C687 P59287 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 455296 T2086 C687 P59287 oasc.SyncStrategy.syncToMe http://127.0.0.1:59287/ul/ki/collection1/ has no replicas
>   [junit4]   2> 455296 T2086 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59287/ul/ki/collection1/ shard1
>   [junit4]   2> 455298 T2086 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
>   [junit4]   2> 455304 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 455319 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 455321 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 455322 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 455339 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 455349 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 455456 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 455458 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 455480 T2086 oasc.ZkController.register We are http://127.0.0.1:59287/ul/ki/collection1/ and leader is http://127.0.0.1:59287/ul/ki/collection1/
>   [junit4]   2> 455481 T2086 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59287/ul/ki
>   [junit4]   2> 455481 T2086 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>   [junit4]   2> 455481 T2086 oasc.ZkController.publish publishing core=collection1 state=active
>   [junit4]   2> 455481 T2086 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 455485 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 455485 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 455485 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 455488 T2086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 455492 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 455494 T2080 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"active",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:59287/ul/ki",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:59287_ul%2Fki",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"control_collection",
>   [junit4]   2> 	  "numShards":"3",
>   [junit4]   2> 	  "core_node_name":"core_node1"}
>   [junit4]   2> 455503 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 455611 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 455612 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>   [junit4]   2> 455713 T2057 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
>   [junit4]   2> 455714 T2057 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 455718 T2057 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59291
>   [junit4]   2> 455718 T2057 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 455718 T2057 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 455719 T2057 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321
>   [junit4]   2> 455719 T2057 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/'
>   [junit4]   2> 455747 T2057 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/solr.xml
>   [junit4]   2> 455841 T2057 oasc.CoreContainer.<init> New CoreContainer 2009305472
>   [junit4]   2> 455842 T2057 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/]
>   [junit4]   2> 455843 T2057 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 455843 T2057 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 455844 T2057 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 455844 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 455844 T2057 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 455844 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 455845 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 455845 T2057 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 455846 T2057 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 455859 T2057 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 455859 T2057 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 455860 T2057 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 455861 T2057 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59284/solr
>   [junit4]   2> 455861 T2057 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 455863 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 455867 T2099 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4885a7ed name:ZooKeeperConnection Watcher:127.0.0.1:59284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 455868 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 455876 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 455881 T2101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@523e646e name:ZooKeeperConnection Watcher:127.0.0.1:59284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 455882 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 455893 T2057 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 456904 T2057 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59291_ul%2Fki
>   [junit4]   2> 456906 T2057 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59291_ul%2Fki
>   [junit4]   2> 456913 T2079 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 456914 T2101 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 456914 T2088 oascc.ZkStateReader$3.process Updating live nodes... (2)
>   [junit4]   2> 456933 T2102 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 456933 T2102 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 456936 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 456936 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 456936 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 456936 T2102 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 456941 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 456942 T2080 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:59291/ul/ki",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:59291_ul%2Fki",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"3",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 456942 T2080 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
>   [junit4]   2> 456943 T2080 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
>   [junit4]   2> 456950 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 457058 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 457058 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 457058 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 457938 T2102 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 457939 T2102 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/collection1
>   [junit4]   2> 457939 T2102 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
>   [junit4]   2> 457941 T2102 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 457941 T2102 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
>   [junit4]   2> 457945 T2102 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
>   [junit4]   2> 457945 T2102 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/collection1/'
>   [junit4]   2> 457947 T2102 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/collection1/lib/classes/' to classloader
>   [junit4]   2> 457947 T2102 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/collection1/lib/README' to classloader
>   [junit4]   2> 458023 T2102 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 458104 T2102 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 458107 T2102 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 458121 T2102 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 458706 T2102 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 458709 T2102 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 458720 T2102 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 458728 T2102 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 458803 T2102 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>   [junit4]   2> 458805 T2102 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392323941321/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty1/
>   [junit4]   2> 458805 T2102 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b3d4e26
>   [junit4]   2> 458806 T2102 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty1
>   [junit4]   2> 458806 T2102 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty1/index/
>   [junit4]   2> 458807 T2102 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty1/index' doesn't exist. Creating new index...
>   [junit4]   2> 458807 T2102 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty1/index
>   [junit4]   2> 458808 T2102 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5503282300917641]
>   [junit4]   2> 458811 T2102 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty1/index,segFN=segments_1,generation=1}
>   [junit4]   2> 458811 T2102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 458816 T2102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 458817 T2102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 458817 T2102 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 458817 T2102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 458818 T2102 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 458818 T2102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 458819 T2102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 458819 T2102 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 458819 T2102 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 458821 T2102 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 458822 T2102 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 458823 T2102 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 458823 T2102 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 458825 T2102 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 458826 T2102 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 458827 T2102 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 458844 T2102 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 458850 T2102 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 458850 T2102 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 458851 T2102 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=36.1337890625, floorSegmentMB=1.6708984375, forceMergeDeletesPctAllowed=6.1288322191713505, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>   [junit4]   2> 458854 T2102 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty1/index,segFN=segments_1,generation=1}
>   [junit4]   2> 458854 T2102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 458854 T2102 oass.SolrIndexSearcher.<init> Opening Searcher@7f90b2b2[collection1] main
>   [junit4]   2> 458862 T2103 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f90b2b2[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 458866 T2102 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 458868 T2057 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 458868 T2057 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 458868 T2106 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59291/ul/ki collection:collection1 shard:shard2
>   [junit4]   2> 458871 T2106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
>   [junit4]   2> 458886 T2106 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
>   [junit4]   2> 458891 T2106 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 458891 T2106 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C688 name=collection1 org.apache.solr.core.SolrCore@16981428 url=http://127.0.0.1:59291/ul/ki/collection1 node=127.0.0.1:59291_ul%2Fki C688_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:59291/ul/ki, core=collection1, node_name=127.0.0.1:59291_ul%2Fki}
>   [junit4]   2> 458891 T2106 C688 P59291 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59291/ul/ki/collection1/
>   [junit4]   2> 458891 T2106 C688 P59291 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 458892 T2106 C688 P59291 oasc.SyncStrategy.syncToMe http://127.0.0.1:59291/ul/ki/collection1/ has no replicas
>   [junit4]   2> 458892 T2106 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59291/ul/ki/collection1/ shard2
>   [junit4]   2> 458892 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 458893 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 458893 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 458892 T2106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
>   [junit4]   2> 458900 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 458911 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 458915 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 458916 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 458916 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 458927 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 458933 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 459041 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 459042 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 459043 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 459068 T2106 oasc.ZkController.register We are http://127.0.0.1:59291/ul/ki/collection1/ and leader is http://127.0.0.1:59291/ul/ki/collection1/
>   [junit4]   2> 459068 T2106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59291/ul/ki
>   [junit4]   2> 459068 T2106 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>   [junit4]   2> 459068 T2106 oasc.ZkController.publish publishing core=collection1 state=active
>   [junit4]   2> 459068 T2106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 459070 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 459071 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 459071 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 459073 T2106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 459076 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 459078 T2080 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"active",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:59291/ul/ki",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:59291_ul%2Fki",
>   [junit4]   2> 	  "shard":"shard2",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"3",
>   [junit4]   2> 	  "core_node_name":"core_node1"}
>   [junit4]   2> 459083 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 459190 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 459191 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 459194 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>   [junit4]   2> 459265 T2057 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
>   [junit4]   2> 459266 T2057 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 459272 T2057 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59294
>   [junit4]   2> 459272 T2057 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 459273 T2057 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 459273 T2057 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908
>   [junit4]   2> 459273 T2057 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/'
>   [junit4]   2> 459302 T2057 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/solr.xml
>   [junit4]   2> 459392 T2057 oasc.CoreContainer.<init> New CoreContainer 55012847
>   [junit4]   2> 459392 T2057 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/]
>   [junit4]   2> 459394 T2057 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 459394 T2057 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 459394 T2057 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 459394 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 459395 T2057 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 459395 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 459395 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 459396 T2057 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 459396 T2057 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 459406 T2057 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 459406 T2057 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 459407 T2057 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 459407 T2057 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59284/solr
>   [junit4]   2> 459407 T2057 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 459409 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 459413 T2117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46826797 name:ZooKeeperConnection Watcher:127.0.0.1:59284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 459414 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 459421 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 459426 T2119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33f387c8 name:ZooKeeperConnection Watcher:127.0.0.1:59284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 459427 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 459444 T2057 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 460458 T2057 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59294_ul%2Fki
>   [junit4]   2> 460461 T2057 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59294_ul%2Fki
>   [junit4]   2> 460469 T2101 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 460470 T2088 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 460470 T2119 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 460470 T2079 oascc.ZkStateReader$3.process Updating live nodes... (3)
>   [junit4]   2> 460488 T2120 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 460489 T2120 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 460492 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 460493 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 460493 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 460492 T2120 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 460498 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 460499 T2080 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:59294/ul/ki",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:59294_ul%2Fki",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"3",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 460499 T2080 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
>   [junit4]   2> 460500 T2080 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
>   [junit4]   2> 460507 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 460615 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 460615 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 460615 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 460615 T2119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 461496 T2120 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 461496 T2120 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/collection1
>   [junit4]   2> 461497 T2120 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
>   [junit4]   2> 461498 T2120 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 461499 T2120 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
>   [junit4]   2> 461502 T2120 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
>   [junit4]   2> 461503 T2120 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/collection1/'
>   [junit4]   2> 461505 T2120 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/collection1/lib/classes/' to classloader
>   [junit4]   2> 461506 T2120 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/collection1/lib/README' to classloader
>   [junit4]   2> 461596 T2120 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 461695 T2120 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 461699 T2120 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 461712 T2120 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 462289 T2120 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 462292 T2120 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 462301 T2120 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 462310 T2120 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 462382 T2120 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>   [junit4]   2> 462383 T2120 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392323944908/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty2/
>   [junit4]   2> 462384 T2120 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b3d4e26
>   [junit4]   2> 462385 T2120 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty2
>   [junit4]   2> 462385 T2120 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty2/index/
>   [junit4]   2> 462386 T2120 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty2/index' doesn't exist. Creating new index...
>   [junit4]   2> 462386 T2120 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty2/index
>   [junit4]   2> 462387 T2120 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5503282300917641]
>   [junit4]   2> 462390 T2120 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty2/index,segFN=segments_1,generation=1}
>   [junit4]   2> 462390 T2120 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 462395 T2120 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 462395 T2120 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 462396 T2120 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 462396 T2120 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 462396 T2120 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 462397 T2120 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 462397 T2120 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 462398 T2120 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 462398 T2120 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 462398 T2120 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 462399 T2120 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 462399 T2120 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 462400 T2120 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 462400 T2120 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 462401 T2120 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 462402 T2120 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 462411 T2120 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 462415 T2120 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 462415 T2120 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 462416 T2120 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=36.1337890625, floorSegmentMB=1.6708984375, forceMergeDeletesPctAllowed=6.1288322191713505, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>   [junit4]   2> 462418 T2120 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty2/index,segFN=segments_1,generation=1}
>   [junit4]   2> 462418 T2120 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 462418 T2120 oass.SolrIndexSearcher.<init> Opening Searcher@1d0cd764[collection1] main
>   [junit4]   2> 462765 T2121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d0cd764[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
>   [junit4]   2> 462765 T2120 oasc.CoreContainer.registerCore registering core: collection1
>   [junit4]   2> 462767 T2057 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
>   [junit4]   2> 462767 T2057 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
>   [junit4]   2> 462767 T2124 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59294/ul/ki collection:collection1 shard:shard1
>   [junit4]   2> 462774 T2124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
>   [junit4]   2> 462790 T2124 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
>   [junit4]   2> 462804 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 462805 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 462806 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 462807 T2124 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
>   [junit4]   2> 462808 T2124 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
>   [junit4]   2> ASYNC  NEW_CORE C689 name=collection1 org.apache.solr.core.SolrCore@46ae6cee url=http://127.0.0.1:59294/ul/ki/collection1 node=127.0.0.1:59294_ul%2Fki C689_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:59294/ul/ki, core=collection1, node_name=127.0.0.1:59294_ul%2Fki}
>   [junit4]   2> 462808 T2124 C689 P59294 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59294/ul/ki/collection1/
>   [junit4]   2> 462808 T2124 C689 P59294 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
>   [junit4]   2> 462809 T2124 C689 P59294 oasc.SyncStrategy.syncToMe http://127.0.0.1:59294/ul/ki/collection1/ has no replicas
>   [junit4]   2> 462809 T2124 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59294/ul/ki/collection1/ shard1
>   [junit4]   2> 462809 T2124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
>   [junit4]   2> 462813 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 462827 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 462831 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462831 T2119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462832 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462832 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462847 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 462856 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 462962 T2119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462962 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462964 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462964 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 462994 T2124 oasc.ZkController.register We are http://127.0.0.1:59294/ul/ki/collection1/ and leader is http://127.0.0.1:59294/ul/ki/collection1/
>   [junit4]   2> 462994 T2124 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59294/ul/ki
>   [junit4]   2> 462995 T2124 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
>   [junit4]   2> 462995 T2124 oasc.ZkController.publish publishing core=collection1 state=active
>   [junit4]   2> 462995 T2124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 462998 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 462999 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 462999 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 463000 T2124 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 463005 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 463007 T2080 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"active",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:59294/ul/ki",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:59294_ul%2Fki",
>   [junit4]   2> 	  "shard":"shard1",
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"3",
>   [junit4]   2> 	  "core_node_name":"core_node2"}
>   [junit4]   2> 463014 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 463123 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 463123 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 463124 T2119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 463123 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>   [junit4]   2> 463204 T2057 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
>   [junit4]   2> 463205 T2057 oejs.Server.doStart jetty-8.1.10.v20130312
>   [junit4]   2> 463210 T2057 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59297
>   [junit4]   2> 463211 T2057 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
>   [junit4]   2> 463212 T2057 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 463213 T2057 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807
>   [junit4]   2> 463213 T2057 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/'
>   [junit4]   2> 463255 T2057 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/solr.xml
>   [junit4]   2> 463371 T2057 oasc.CoreContainer.<init> New CoreContainer 1156457194
>   [junit4]   2> 463372 T2057 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/]
>   [junit4]   2> 463374 T2057 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
>   [junit4]   2> 463374 T2057 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
>   [junit4]   2> 463374 T2057 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
>   [junit4]   2> 463375 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
>   [junit4]   2> 463375 T2057 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
>   [junit4]   2> 463375 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
>   [junit4]   2> 463375 T2057 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
>   [junit4]   2> 463375 T2057 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
>   [junit4]   2> 463376 T2057 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
>   [junit4]   2> 463386 T2057 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 463387 T2057 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 463387 T2057 oasc.CoreContainer.load Host Name: 127.0.0.1
>   [junit4]   2> 463387 T2057 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59284/solr
>   [junit4]   2> 463388 T2057 oasc.ZkController.checkChrootPath zkHost includes chroot
>   [junit4]   2> 463389 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 463394 T2135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e365400 name:ZooKeeperConnection Watcher:127.0.0.1:59284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 463395 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 463405 T2057 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
>   [junit4]   2> 463410 T2137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36aaff68 name:ZooKeeperConnection Watcher:127.0.0.1:59284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 463411 T2057 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
>   [junit4]   2> 463422 T2057 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
>   [junit4]   2> 464432 T2057 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59297_ul%2Fki
>   [junit4]   2> 464435 T2057 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59297_ul%2Fki
>   [junit4]   2> 464442 T2101 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 464443 T2079 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 464444 T2119 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 464444 T2088 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 464444 T2137 oascc.ZkStateReader$3.process Updating live nodes... (4)
>   [junit4]   2> 464464 T2138 oasc.ZkController.publish publishing core=collection1 state=down
>   [junit4]   2> 464464 T2138 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
>   [junit4]   2> 464467 T2138 oasc.ZkController.waitForCoreNodeName look for our core node name
>   [junit4]   2> 464467 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 464467 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 464467 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 464472 T2080 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
>   [junit4]   2> 464474 T2080 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
>   [junit4]   2> 	  "operation":"state",
>   [junit4]   2> 	  "state":"down",
>   [junit4]   2> 	  "base_url":"http://127.0.0.1:59297/ul/ki",
>   [junit4]   2> 	  "core":"collection1",
>   [junit4]   2> 	  "roles":null,
>   [junit4]   2> 	  "node_name":"127.0.0.1:59297_ul%2Fki",
>   [junit4]   2> 	  "shard":null,
>   [junit4]   2> 	  "shard_range":null,
>   [junit4]   2> 	  "shard_state":"active",
>   [junit4]   2> 	  "shard_parent":null,
>   [junit4]   2> 	  "collection":"collection1",
>   [junit4]   2> 	  "numShards":"3",
>   [junit4]   2> 	  "core_node_name":null}
>   [junit4]   2> 464474 T2080 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
>   [junit4]   2> 464475 T2080 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
>   [junit4]   2> 464481 T2079 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>   [junit4]   2> 464589 T2119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>   [junit4]   2> 464590 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>   [junit4]   2> 464590 T2079 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>   [junit4]   2> 464590 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>   [junit4]   2> 464590 T2137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
>   [junit4]   2> 465469 T2138 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
>   [junit4]   2> 465470 T2138 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/collection1
>   [junit4]   2> 465470 T2138 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
>   [junit4]   2> 465472 T2138 oasc.ZkController.createCollectionZkNode Collection zkNode exists
>   [junit4]   2> 465472 T2138 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
>   [junit4]   2> 465475 T2138 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
>   [junit4]   2> 465475 T2138 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/collection1/'
>   [junit4]   2> 465476 T2138 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/collection1/lib/classes/' to classloader
>   [junit4]   2> 465477 T2138 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/collection1/lib/README' to classloader
>   [junit4]   2> 465543 T2138 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
>   [junit4]   2> 465616 T2138 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 465619 T2138 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
>   [junit4]   2> 465632 T2138 oass.IndexSchema.readSchema [collection1] Schema name=test
>   [junit4]   2> 466240 T2138 oass.IndexSchema.readSchema default search field in schema is text
>   [junit4]   2> 466243 T2138 oass.IndexSchema.readSchema unique key field: id
>   [junit4]   2> 466253 T2138 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 466260 T2138 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
>   [junit4]   2> 466343 T2138 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
>   [junit4]   2> 466343 T2138 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392323948807/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty3/
>   [junit4]   2> 466344 T2138 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b3d4e26
>   [junit4]   2> 466345 T2138 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty3
>   [junit4]   2> 466346 T2138 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty3/index/
>   [junit4]   2> 466348 T2138 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty3/index' doesn't exist. Creating new index...
>   [junit4]   2> 466349 T2138 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty3/index
>   [junit4]   2> 466349 T2138 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5503282300917641]
>   [junit4]   2> 466354 T2138 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty3/index,segFN=segments_1,generation=1}
>   [junit4]   2> 466355 T2138 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 466362 T2138 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
>   [junit4]   2> 466363 T2138 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
>   [junit4]   2> 466364 T2138 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
>   [junit4]   2> 466364 T2138 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 466364 T2138 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
>   [junit4]   2> 466365 T2138 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>   [junit4]   2> 466365 T2138 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 466366 T2138 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>   [junit4]   2> 466366 T2138 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
>   [junit4]   2> 466367 T2138 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
>   [junit4]   2> 466368 T2138 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
>   [junit4]   2> 466368 T2138 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
>   [junit4]   2> 466369 T2138 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
>   [junit4]   2> 466369 T2138 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
>   [junit4]   2> 466370 T2138 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
>   [junit4]   2> 466371 T2138 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
>   [junit4]   2> 466385 T2138 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>   [junit4]   2> 466391 T2138 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>   [junit4]   2> 466392 T2138 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>   [junit4]   2> 466395 T2138 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=36.1337890625, floorSegmentMB=1.6708984375, forceMergeDeletesPctAllowed=6.1288322191713505, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>   [junit4]   2> 466398 T2138 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
>   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392323938382/jetty3/index,segFN=segments_1,generation=1}
>   [junit4]   2> 466398 T2138 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
>   [junit4]   2> 466398 T2138 oass.SolrIndexSearcher.<i
> 
> [...truncated too long message...]
> 
>   2> 	  58) Thread[id=2644, name=HashSessionScavenger-86, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
>   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
>   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
>   [junit4]   2> 	  59) Thread[id=2606, name=qtp884369320-2606 Acceptor0 SelectChannelConnector@127.0.0.1:59300, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
>   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> 	  60) Thread[id=2218, name=Thread-953, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
>   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
>   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1018)
>   [junit4]   2> 	  61) Thread[id=2596, name=Thread-1167, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
>   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
>   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1018)
>   [junit4]   2> 	  62) Thread[id=2636, name=Thread-1016-SendThread(localhost:59284), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
>   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
>   [junit4]   2> 	  63) Thread[id=2684, name=Thread-1220, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
>   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
>   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1018)
>   [junit4]   2> 	  64) Thread[id=2621, name=Thread-1183, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
>   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
>   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:1018)
>   [junit4]   2> 	  65) Thread[id=2660, name=searcherExecutor-1605-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
>   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> 	  66) Thread[id=2663, name=coreZkRegister-1601-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
>   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
>   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
>   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
>   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> 	  67) Thread[id=2591, name=Thread-1016-SendThread(localhost:59284), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
>   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
>   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
>   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
>   [junit4]   2> 	  68) Thread[id=2126, name=qtp1292585257-2126 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
>   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
>   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
>   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
>   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
>   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
>   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
>   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
>   [junit4]   2> 	  69) Thread[id=2615, name=Thread-1016-SendThread(localhost:59284), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
>   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
>   [junit4]   2> 	  70) Thread[id=2125, name=HashSessionScavenger-65, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
>   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
>   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:552)
>   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
>   [junit4]   2> NOTE: test params are: codec=Lucene46: {rnd_b=Pulsing41(freqCutoff=12 minBlockSize=52 maxBlockSize=189), a_t=MockFixedIntBlock(blockSize=1812), _version_=MockVariableIntBlock(baseBlockSize=44), id=Pulsing41(freqCutoff=12 minBlockSize=52 maxBlockSize=189), a_si=MockVariableIntBlock(baseBlockSize=44)}, docValues:{}, sim=DefaultSimilarity, locale=en_NZ, timezone=Indian/Comoro
>   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=82,free=52050784,total=188837888
>   [junit4]   2> NOTE: All tests run in this JVM: [CSVRequestHandlerTest, TestReplicationHandler, TestTrie, CollectionsAPIDistributedZkTest, DirectSolrConnectionTest, TestArbitraryIndexDir, CopyFieldTest, TestFieldTypeResource, OverseerTest, TestCSVLoader, ShowFileRequestHandlerTest, HardAutoCommitTest, SpatialFilterTest, TestCursorMarkWithoutUniqueKey, StandardRequestHandlerTest, PeerSyncTest, MultiTermTest, CursorPagingTest, TestWriterPerf, ParsingFieldUpdateProcessorsTest, SystemInfoHandlerTest, ExternalFileFieldSortTest, TestBM25SimilarityFactory, TestRandomMergePolicy, TestJoin, TestQuerySenderListener, StatsComponentTest, TestSimpleQParserPlugin, DOMUtilTest, SOLR749Test, TestFastOutputStream, SolrTestCaseJ4Test, TestSolrCoreProperties, BadCopyFieldTest, TestElisionMultitermQuery, DisMaxRequestHandlerTest, DistanceFunctionTest, LeaderElectionTest, EchoParamsTest, RAMDirectoryFactoryTest, ChaosMonkeySafeLeaderTest]
>   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=7942C2603C4247B0 -Dtests.slow=true -Dtests.locale=en_NZ -Dtests.timezone=Indian/Comoro -Dtests.file.encoding=ISO-8859-1
>   [junit4] ERROR   0.00s | ChaosMonkeySafeLeaderTest (suite) <<<
>   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7942C2603C4247B0]:0)
>   [junit4] Completed in 7229.33s, 1 test, 2 errors <<< FAILURES!
> 
> [...truncated 967 lines...]
> BUILD FAILED
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1308: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:941: There were test failures: 362 suites, 1562 tests, 1 suite-level error, 1 error, 1314 ignored
> 
> Total time: 161 minutes 51 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message