lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 2028 - Still Failing!
Date Thu, 01 Jan 2015 08:32:24 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/2028/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC (asserts: true)

2 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([ACA3BF961C3082C4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([ACA3BF961C3082C4]:0)




Build Log:
[...truncated 9425 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/init-core-data-001
   [junit4]   2> 2915835 T14355 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2915836 T14355 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2915852 T14355 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2915853 T14355 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2915855 T14356 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2915956 T14355 oasc.ZkTestServer.run start zk server on port:53946
   [junit4]   2> 2915957 T14355 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2915958 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2915965 T14363 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52e17cd5 name:ZooKeeperConnection Watcher:127.0.0.1:53946 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2915966 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2915967 T14355 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2915967 T14355 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2915993 T14355 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2915995 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2916002 T14366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bb87f7f name:ZooKeeperConnection Watcher:127.0.0.1:53946/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2916003 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2916003 T14355 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2916004 T14355 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2916019 T14355 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2916030 T14355 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2916041 T14355 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2916052 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2916054 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2916068 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2916069 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2916079 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2916080 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2916090 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2916092 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2916101 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2916103 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2916112 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2916112 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2916119 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2916120 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2916135 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2916136 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2916146 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2916147 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2916156 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2916157 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2916169 T14355 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2916170 T14355 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2918430 T14355 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2918439 T14355 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53949
   [junit4]   2> 2918441 T14355 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4554617c
   [junit4]   2> 2918442 T14355 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2918443 T14355 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002
   [junit4]   2> 2918444 T14355 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/'
   [junit4]   2> 2918503 T14355 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/solr.xml
   [junit4]   2> 2918524 T14355 oasc.CoreContainer.<init> New CoreContainer 1170164933
   [junit4]   2> 2918525 T14355 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/]
   [junit4]   2> 2918525 T14355 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2918527 T14355 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2918527 T14355 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2918527 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2918528 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2918529 T14355 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2918529 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2918529 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2918530 T14355 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2918530 T14355 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2918531 T14355 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2918532 T14355 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2918533 T14355 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2918533 T14355 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2918534 T14355 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53946/solr
   [junit4]   2> 2918534 T14355 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2918534 T14355 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2918536 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2918539 T14378 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ed58da8 name:ZooKeeperConnection Watcher:127.0.0.1:53946 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2918539 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2918540 T14355 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2918545 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2918549 T14381 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b336c6a name:ZooKeeperConnection Watcher:127.0.0.1:53946/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2918549 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2918553 T14355 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2918564 T14355 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2918573 T14355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2918584 T14355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2918593 T14355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2918602 T14355 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2918607 T14355 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53949_
   [junit4]   2> 2918610 T14355 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53949_
   [junit4]   2> 2918619 T14355 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2918625 T14355 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2918631 T14355 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2918638 T14355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53949_
   [junit4]   2> 2918638 T14355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2918645 T14355 oasc.Overseer.start Overseer (id=93067262210670595-127.0.0.1:53949_-n_0000000000) starting
   [junit4]   2> 2918652 T14355 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2918682 T14355 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2918684 T14383 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2918685 T14355 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2918693 T14355 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2918700 T14355 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2918715 T14382 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2918722 T14385 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2918723 T14385 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2918725 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2918733 T14385 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2918735 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53949",
   [junit4]   2> 	  "node_name":"127.0.0.1:53949_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 2918737 T14382 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53949",
   [junit4]   2> 	  "node_name":"127.0.0.1:53949_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2918738 T14382 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 2918738 T14382 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2918742 T14381 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2919734 T14385 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2919735 T14385 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2919737 T14385 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2919738 T14385 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2919743 T14385 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2919744 T14385 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/collection1/'
   [junit4]   2> 2919745 T14385 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2919746 T14385 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2919747 T14385 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2919785 T14385 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 2919791 T14385 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 2919803 T14385 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 2919807 T14385 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2919818 T14385 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2919821 T14385 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2919827 T14385 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2919963 T14385 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2919965 T14385 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2919965 T14385 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2919971 T14385 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2920022 T14385 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2920022 T14385 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2920022 T14385 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-002/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/control/data/
   [junit4]   2> 2920023 T14385 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8ad79f4
   [junit4]   2> 2920025 T14385 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/control/data
   [junit4]   2> 2920025 T14385 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/control/data/index/
   [junit4]   2> 2920026 T14385 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2920027 T14385 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/control/data/index
   [junit4]   2> 2920027 T14385 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=87.568359375, floorSegmentMB=1.501953125, forceMergeDeletesPctAllowed=27.55218996331614, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8881378966558299
   [junit4]   2> 2920030 T14385 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2920030 T14385 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2920040 T14385 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2920041 T14385 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2920042 T14385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2920043 T14385 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2920043 T14385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2920043 T14385 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2920044 T14385 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2920045 T14385 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2920045 T14385 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2920046 T14385 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2920047 T14385 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2920047 T14385 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2920047 T14385 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2920048 T14385 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 2920048 T14385 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 2920049 T14385 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 2920049 T14385 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 2920050 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 2920051 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 2920051 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 2920053 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 2920053 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 2920054 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 2920054 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 2920055 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 2920055 T14385 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2920056 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2920057 T14385 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2920064 T14385 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2920066 T14385 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2920067 T14385 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2920068 T14385 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2920069 T14385 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 2920071 T14385 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2920071 T14385 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2920072 T14385 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=47.3642578125, floorSegmentMB=0.8271484375, forceMergeDeletesPctAllowed=23.156128234359365, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2920075 T14385 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2920075 T14385 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2920075 T14385 oass.SolrIndexSearcher.<init> Opening Searcher@6c5a0de0[collection1] main
   [junit4]   2> 2920078 T14385 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2920082 T14385 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2920082 T14385 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2920083 T14385 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2920084 T14385 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2920084 T14385 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2920086 T14385 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2920086 T14385 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2920086 T14385 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2920086 T14385 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 2920087 T14385 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2920088 T14386 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6c5a0de0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2920088 T14389 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53949 collection:control_collection shard:shard1
   [junit4]   2> 2920089 T14355 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2920090 T14355 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2920091 T14355 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2920092 T14389 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2920093 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2920097 T14392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22de1753 name:ZooKeeperConnection Watcher:127.0.0.1:53946/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2920098 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2920098 T14355 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2920101 T14355 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2920107 T14389 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2920110 T14355 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2920111 T14355 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 2920111 T14355 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2920112 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2920114 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2920117 T14389 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2920117 T14389 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3653 name=collection1 org.apache.solr.core.SolrCore@401a060f url=http://127.0.0.1:53949/collection1 node=127.0.0.1:53949_ C3653_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:53949, node_name=127.0.0.1:53949_, state=down}
   [junit4]   2> 2920117 T14389 C3653 P53949 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53949/collection1/
   [junit4]   2> 2920118 T14389 C3653 P53949 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2920119 T14389 C3653 P53949 oasc.SyncStrategy.syncToMe http://127.0.0.1:53949/collection1/ has no replicas
   [junit4]   2> 2920119 T14389 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53949/collection1/ shard1
   [junit4]   2> 2920118 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 2920120 T14389 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2920119 T14395 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76d3574e name:ZooKeeperConnection Watcher:127.0.0.1:53946/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2920122 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2920123 T14355 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2920133 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "stateFormat":2} current state version: 1
   [junit4]   2> 2920134 T14382 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 2920136 T14392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2920140 T14381 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2920166 T14396 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2920193 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53949",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 2
   [junit4]   2> 2920194 T14382 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 2920198 T14392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2920215 T14396 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2920219 T14392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2920286 T14389 oasc.ZkController.register We are http://127.0.0.1:53949/collection1/ and leader is http://127.0.0.1:53949/collection1/
   [junit4]   2> 2920288 T14389 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53949
   [junit4]   2> 2920288 T14389 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2920288 T14389 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2920288 T14389 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2920291 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2920293 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53949",
   [junit4]   2> 	  "node_name":"127.0.0.1:53949_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 2920294 T14382 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53949",
   [junit4]   2> 	  "node_name":"127.0.0.1:53949_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2920404 T14381 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2920405 T14392 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2921809 T14355 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2921810 T14355 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2921816 T14355 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53954
   [junit4]   2> 2921818 T14355 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4554617c
   [junit4]   2> 2921819 T14355 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2921819 T14355 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003
   [junit4]   2> 2921819 T14355 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/'
   [junit4]   2> 2921874 T14355 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/solr.xml
   [junit4]   2> 2921925 T14355 oasc.CoreContainer.<init> New CoreContainer 850377707
   [junit4]   2> 2921926 T14355 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/]
   [junit4]   2> 2921927 T14355 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2921928 T14355 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2921928 T14355 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2921929 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2921929 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2921930 T14355 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2921931 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2921931 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2921932 T14355 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2921932 T14355 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2921933 T14355 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2921934 T14355 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2921934 T14355 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2921935 T14355 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2921936 T14355 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53946/solr
   [junit4]   2> 2921937 T14355 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2921938 T14355 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2921939 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2921944 T14408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4167548d name:ZooKeeperConnection Watcher:127.0.0.1:53946 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2921944 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2921945 T14355 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2921950 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2921954 T14411 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70150838 name:ZooKeeperConnection Watcher:127.0.0.1:53946/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2921954 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2921973 T14355 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2922992 T14355 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53954_
   [junit4]   2> 2922994 T14355 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53954_
   [junit4]   2> 2923005 T14355 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2923022 T14412 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2923022 T14412 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2923026 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2923029 T14412 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 2923029 T14412 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 2923029 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53954",
   [junit4]   2> 	  "node_name":"127.0.0.1:53954_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 2923031 T14412 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 2923031 T14382 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53954",
   [junit4]   2> 	  "node_name":"127.0.0.1:53954_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2923032 T14382 oasco.ReplicaMutator.updateState Collection already exists with numShards=3
   [junit4]   2> 2923033 T14382 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2923035 T14412 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 2923035 T14412 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2923036 T14382 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 2923038 T14411 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2923039 T14411 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 2924036 T14412 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2924037 T14412 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2924039 T14412 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2924040 T14412 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2924046 T14412 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2924046 T14412 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/collection1/'
   [junit4]   2> 2924048 T14412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2924051 T14412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2924051 T14412 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2924121 T14412 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 2924129 T14412 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 2924148 T14412 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 2924154 T14412 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2924165 T14412 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2924168 T14412 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2924174 T14412 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2924294 T14412 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2924297 T14412 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2924299 T14412 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2924314 T14412 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2924392 T14412 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2924393 T14412 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2924393 T14412 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-003/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty1/
   [junit4]   2> 2924393 T14412 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8ad79f4
   [junit4]   2> 2924394 T14412 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty1
   [junit4]   2> 2924395 T14412 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty1/index/
   [junit4]   2> 2924396 T14412 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2924396 T14412 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty1/index
   [junit4]   2> 2924397 T14412 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=87.568359375, floorSegmentMB=1.501953125, forceMergeDeletesPctAllowed=27.55218996331614, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8881378966558299
   [junit4]   2> 2924401 T14412 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2924401 T14412 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2924408 T14412 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2924409 T14412 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2924410 T14412 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2924410 T14412 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2924411 T14412 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2924411 T14412 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2924412 T14412 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2924413 T14412 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2924413 T14412 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2924414 T14412 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2924414 T14412 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2924414 T14412 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2924415 T14412 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2924416 T14412 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 2924416 T14412 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 2924418 T14412 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 2924419 T14412 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 2924420 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 2924425 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 2924430 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 2924431 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 2924432 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 2924433 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 2924434 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 2924435 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 2924436 T14412 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2924436 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2924437 T14412 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2924449 T14412 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2924454 T14412 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2924455 T14412 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2924456 T14412 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2924456 T14412 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 2924459 T14412 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2924460 T14412 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2924460 T14412 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=47.3642578125, floorSegmentMB=0.8271484375, forceMergeDeletesPctAllowed=23.156128234359365, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2924461 T14412 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2924462 T14412 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2924462 T14412 oass.SolrIndexSearcher.<init> Opening Searcher@ea45e9[collection1] main
   [junit4]   2> 2924462 T14412 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2924465 T14412 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2924465 T14412 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2924467 T14412 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2924467 T14412 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2924467 T14412 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2924469 T14412 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2924469 T14412 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2924469 T14412 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2924471 T14412 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 2924472 T14413 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ea45e9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2924473 T14412 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2924475 T14416 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53954 collection:collection1 shard:shard2
   [junit4]   2> 2924477 T14355 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2924478 T14355 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2924479 T14416 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2924499 T14416 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2924505 T14416 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2924506 T14416 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3654 name=collection1 org.apache.solr.core.SolrCore@21720329 url=http://127.0.0.1:53954/collection1 node=127.0.0.1:53954_ C3654_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:53954, node_name=127.0.0.1:53954_, state=down}
   [junit4]   2> 2924506 T14416 C3654 P53954 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53954/collection1/
   [junit4]   2> 2924507 T14416 C3654 P53954 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2924508 T14416 C3654 P53954 oasc.SyncStrategy.syncToMe http://127.0.0.1:53954/collection1/ has no replicas
   [junit4]   2> 2924509 T14416 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53954/collection1/ shard2
   [junit4]   2> 2924509 T14416 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2924508 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2924513 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 5
   [junit4]   2> 2924518 T14382 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 2924523 T14411 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2924533 T14411 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 2924567 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2924573 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53954",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 5
   [junit4]   2> 2924577 T14382 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
   [junit4]   2> 2924580 T14411 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2924585 T14411 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 2924623 T14416 oasc.ZkController.register We are http://127.0.0.1:53954/collection1/ and leader is http://127.0.0.1:53954/collection1/
   [junit4]   2> 2924625 T14416 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53954
   [junit4]   2> 2924625 T14416 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2924626 T14416 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2924626 T14416 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2924633 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2924640 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53954",
   [junit4]   2> 	  "node_name":"127.0.0.1:53954_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 2924643 T14382 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53954",
   [junit4]   2> 	  "node_name":"127.0.0.1:53954_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2924650 T14382 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
   [junit4]   2> 2924657 T14411 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2924658 T14416 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 2924661 T14411 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 2926761 T14355 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2926762 T14355 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2926778 T14355 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53957
   [junit4]   2> 2926779 T14355 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4554617c
   [junit4]   2> 2926780 T14355 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2926781 T14355 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004
   [junit4]   2> 2926782 T14355 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/'
   [junit4]   2> 2926870 T14355 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/solr.xml
   [junit4]   2> 2926900 T14355 oasc.CoreContainer.<init> New CoreContainer 1705032371
   [junit4]   2> 2926904 T14355 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/]
   [junit4]   2> 2926905 T14355 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2926906 T14355 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2926907 T14355 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2926907 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2926908 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 2926909 T14355 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2926909 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2926910 T14355 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2926911 T14355 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2926912 T14355 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2926912 T14355 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2926914 T14355 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2926914 T14355 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2926915 T14355 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2926916 T14355 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53946/solr
   [junit4]   2> 2926916 T14355 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2926917 T14355 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2926919 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2926924 T14428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4cd47ece name:ZooKeeperConnection Watcher:127.0.0.1:53946 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2926925 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2926925 T14355 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2926931 T14355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2926944 T14431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@253819cb name:ZooKeeperConnection Watcher:127.0.0.1:53946/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2926944 T14355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2926988 T14355 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2928038 T14355 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53957_
   [junit4]   2> 2928040 T14355 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53957_
   [junit4]   2> 2928059 T14355 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2928083 T14432 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2928083 T14432 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2928088 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2928091 T14432 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 2928091 T14432 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 2928092 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53957",
   [junit4]   2> 	  "node_name":"127.0.0.1:53957_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 2928097 T14432 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 2928097 T14382 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53957",
   [junit4]   2> 	  "node_name":"127.0.0.1:53957_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2928099 T14382 oasco.ReplicaMutator.updateState Collection already exists with numShards=3
   [junit4]   2> 2928099 T14382 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2928102 T14432 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 2928104 T14432 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2928105 T14382 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
   [junit4]   2> 2928107 T14431 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2928107 T14411 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2928113 T14431 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 2928115 T14411 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 2929105 T14432 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2929106 T14432 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2929109 T14432 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2929109 T14432 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2929114 T14432 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2929115 T14432 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/collection1/'
   [junit4]   2> 2929117 T14432 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2929118 T14432 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2929119 T14432 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2929161 T14432 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 2929167 T14432 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 2929184 T14432 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 2929191 T14432 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2929207 T14432 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2929211 T14432 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 2929221 T14432 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2929417 T14432 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2929418 T14432 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2929419 T14432 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2929428 T14432 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2929513 T14432 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2929513 T14432 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2929514 T14432 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-004/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty2/
   [junit4]   2> 2929514 T14432 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@8ad79f4
   [junit4]   2> 2929516 T14432 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty2
   [junit4]   2> 2929517 T14432 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty2/index/
   [junit4]   2> 2929518 T14432 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2929519 T14432 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty2/index
   [junit4]   2> 2929520 T14432 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=87.568359375, floorSegmentMB=1.501953125, forceMergeDeletesPctAllowed=27.55218996331614, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8881378966558299
   [junit4]   2> 2929527 T14432 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2929527 T14432 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2929537 T14432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2929539 T14432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2929539 T14432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2929539 T14432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2929540 T14432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2929540 T14432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2929542 T14432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2929544 T14432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2929544 T14432 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2929549 T14432 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2929550 T14432 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2929550 T14432 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2929551 T14432 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2929552 T14432 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 2929553 T14432 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 2929558 T14432 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 2929560 T14432 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 2929561 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 2929570 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 2929571 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 2929572 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 2929574 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 2929575 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 2929576 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 2929579 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 2929580 T14432 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2929581 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2929583 T14432 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2929598 T14432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2929602 T14432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2929604 T14432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2929607 T14432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2929608 T14432 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 2929611 T14432 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2929611 T14432 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2929613 T14432 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=47.3642578125, floorSegmentMB=0.8271484375, forceMergeDeletesPctAllowed=23.156128234359365, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2929615 T14432 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-ACA3BF961C3082C4-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2929615 T14432 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2929616 T14432 oass.SolrIndexSearcher.<init> Opening Searcher@2d426fab[collection1] main
   [junit4]   2> 2929617 T14432 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2929619 T14432 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2929620 T14432 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2929623 T14432 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2929623 T14432 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2929624 T14432 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2929626 T14432 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2929627 T14432 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2929627 T14432 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2929628 T14432 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 2929630 T14432 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2929631 T14433 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d426fab[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2929633 T14436 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53957 collection:collection1 shard:shard3
   [junit4]   2> 2929637 T14355 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2929639 T14355 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2929643 T14436 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2929659 T14436 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 2929668 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2929671 T14436 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2929671 T14436 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3655 name=collection1 org.apache.solr.core.SolrCore@1f72145c url=http://127.0.0.1:53957/collection1 node=127.0.0.1:53957_ C3655_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:53957, node_name=127.0.0.1:53957_, state=down}
   [junit4]   2> 2929672 T14436 C3655 P53957 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53957/collection1/
   [junit4]   2> 2929673 T14436 C3655 P53957 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2929675 T14436 C3655 P53957 oasc.SyncStrategy.syncToMe http://127.0.0.1:53957/collection1/ has no replicas
   [junit4]   2> 2929675 T14436 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53957/collection1/ shard3
   [junit4]   2> 2929675 T14436 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2929680 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1"} current state version: 5
   [junit4]   2> 2929691 T14382 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 5
   [junit4]   2> 2929695 T14411 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2929700 T14411 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 2929701 T14431 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2929707 T14431 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 2929719 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53957",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 5
   [junit4]   2> 2929721 T14382 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 6
   [junit4]   2> 2929723 T14411 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2929725 T14411 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 2929730 T14431 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2929733 T14431 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 7 
   [junit4]   2> 2929743 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2929784 T14436 oasc.ZkController.register We are http://127.0.0.1:53957/collection1/ and leader is http://127.0.0.1:53957/collection1/
   [junit4]   2> 2929784 T14436 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53957
   [junit4]   2> 2929785 T14436 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2929785 T14436 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2929785 T14436 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2929788 T14381 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 2929792 T14382 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "rol

[...truncated too long message...]

l.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  36) Thread[id=14434, name=Thread-6175, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:502)
   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
   [junit4]   2> 	  37) Thread[id=14438, name=qtp1677243178-14438 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   [junit4]   2> 	        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   [junit4]   2> 	        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
   [junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {rnd_b=PostingsFormat(name=LuceneFixedGap), _version_=PostingsFormat(name=LuceneVarGapFixedInterval), a_t=Lucene50(blocksize=128), a_i=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, sim=DefaultSimilarity, locale=de_CH, timezone=America/Shiprock
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=2,threads=51,free=214109304,total=358612992
   [junit4]   2> NOTE: All tests run in this JVM: [TestIntervalFaceting, TestCollationField, TestFastWriter, DeleteShardTest, CursorPagingTest, TestRandomDVFaceting, DistributedIntervalFacetingTest, ClusterStateTest, TestQueryTypes, DocumentAnalysisRequestHandlerTest, HighlighterMaxOffsetTest, TestDFRSimilarityFactory, ZkSolrClientTest, TestFieldSortValues, TestManagedSchema, RegexBoostProcessorTest, TestCollationFieldDocValues, CollectionsAPIDistributedZkTest, TestCopyFieldCollectionResource, IndexSchemaRuntimeFieldTest, TestSolr4Spatial, SoftAutoCommitTest, RecoveryZkTest, TestSolrDeletionPolicy2, ZkCLITest, NumericFieldsTest, FastVectorHighlighterTest, TestExpandComponent, TestPostingsSolrHighlighter, TestOmitPositions, TestCloudInspectUtil, TestFieldCollectionResource, TestQuerySenderNoQuery, CacheHeaderTest, TestManagedStopFilterFactory, IndexSchemaTest, DirectSolrSpellCheckerTest, TestReloadAndDeleteDocs, CoreAdminHandlerTest, SSLMigrationTest, SpellCheckComponentTest, TestDefaultStatsCache, CloudExitableDirectoryReaderTest, ChaosMonkeyNothingIsSafeTest, TestDefaultSimilarityFactory, SyncSliceTest, TestSolrJ, RemoteQueryErrorTest, SolrRequestParserTest, TestChildDocTransformer, CurrencyFieldXmlFileTest, TestPHPSerializedResponseWriter, TestFiltering, TestRawResponseWriter, TestDocSet, QueryResultKeyTest, ReplicationFactorTest, DeleteReplicaTest, TestRealTimeGet, SimpleMLTQParserTest, RAMDirectoryFactoryTest, TestRecoveryHdfs, TestAnalyzedSuggestions, TestUpdate, ScriptEngineTest, DeleteInactiveReplicaTest, InfoHandlerTest, ReturnFieldsTest, TestDefaultSearchFieldResource, DistributedSuggestComponentTest, DirectUpdateHandlerOptimizeTest, TestReplicationHandler, PolyFieldTest, TestManagedSchemaFieldResource, ZkControllerTest, ConnectionManagerTest, BasicDistributedZk2Test, TestSolrConfigHandlerCloud, TestClusterStateMutator, TestSurroundQueryParser, TestRequestStatusCollectionAPI, SolrXmlInZkTest, SpatialFilterTest, ExternalFileFieldSortTest, DisMaxRequestHandlerTest, DistributedQueryElevationComponentTest, CopyFieldTest, SolrCoreCheckLockOnStartupTest, TestMiniSolrCloudCluster, TestTolerantSearch, OutOfBoxZkACLAndCredentialsProvidersTest, TestCSVResponseWriter, StatelessScriptUpdateProcessorFactoryTest, TestFunctionQuery, HdfsChaosMonkeySafeLeaderTest, CoreContainerCoreInitFailuresTest, ShowFileRequestHandlerTest, CursorMarkTest, EnumFieldTest, TestConfigReload, TestElisionMultitermQuery, DocValuesMultiTest, URLClassifyProcessorTest, TestShortCircuitedRequests, DocumentBuilderTest, TestStandardQParsers, CachingDirectoryFactoryTest, AtomicUpdatesTest, TestNRTOpen, TestPartialUpdateDeduplication, DistributedFacetPivotLongTailTest, SharedFSAutoReplicaFailoverUtilsTest, TestCloudManagedSchemaConcurrent, SuggesterFSTTest, ClusterStateUpdateTest, TestExactSharedStatsCache, NoCacheHeaderTest, UpdateRequestProcessorFactoryTest, TestCollectionAPI, TestQueryUtils, DocValuesMissingTest, TestSolrConfigHandlerConcurrent, TestStressLucene, TestLFUCache, HdfsWriteToMultipleCollectionsTest, TestIndexSearcher, SimplePostToolTest, TestSolrDeletionPolicy1, TestLeaderElectionZkExpiry, TestTrie, MinimalSchemaTest, SOLR749Test, TestDynamicFieldCollectionResource, TestSolrXmlPersistor, DistributedTermsComponentTest, HttpPartitionTest, TestFuzzyAnalyzedSuggestions, TestManagedSchemaDynamicFieldResource, TestSolrDynamicMBean, AnalysisErrorHandlingTest, SolrIndexSplitterTest, TestNonNRTOpen, TestManagedSynonymFilterFactory, ShardRoutingTest, TestValueSourceCache, SchemaVersionSpecificBehaviorTest, BlockDirectoryTest, TestStressVersions, TestSort, HdfsBasicDistributedZk2Test, TestSystemIdResolver, PeerSyncTest, CoreAdminRequestStatusTest, TestBM25SimilarityFactory, TestSortingResponseWriter, DistributedFacetPivotWhiteBoxTest, HdfsSyncSliceTest, LeaderFailoverAfterPartitionTest, TestDocBasedVersionConstraints, TestLRUStatsCache, HardAutoCommitTest, DateFieldTest, AddSchemaFieldsUpdateProcessorFactoryTest, CoreAdminCreateDiscoverTest, ConvertedLegacyTest, DateMathParserTest, UUIDFieldTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SearchHandlerTest, BadCopyFieldTest, TestRecovery, TestInitQParser, SynonymTokenizerTest, TestLuceneMatchVersion, PreAnalyzedUpdateProcessorTest, TestAddFieldRealTimeGet, SuggesterWFSTTest, TestBinaryField, TestMaxScoreQueryParser, CustomCollectionTest, TestSolrConfigHandler, OverseerRolesTest, FieldMutatingUpdateProcessorTest, SolrInfoMBeanTest, TestCoreDiscovery, TestDistributedGrouping, AlternateDirectoryTest, TestUtils, UnloadDistributedZkTest, PreAnalyzedFieldTest, TestCursorMarkWithoutUniqueKey, RollingRestartTest, DistributedFacetPivotSmallAdvancedTest, TestOverriddenPrefixQueryForCustomFieldType, HdfsCollectionsAPIDistributedZkTest, AnalyticsMergeStrategyTest, TestConfig, ResponseHeaderTest, DistanceFunctionTest, TestOrdValues, SortByFunctionTest, TermsComponentTest, TestFieldResource, TestJoin, RequiredFieldsTest, MergeStrategyTest, TestSearcherReuse, MBeansHandlerTest, TestExceedMaxTermLength, TestQuerySenderListener, TestClassNameShortening, BadComponentTest, TestDistribDocBasedVersion, PathHierarchyTokenizerFactoryTest, DistributedDebugComponentTest, ParsingFieldUpdateProcessorsTest, LegacyHTMLStripCharFilterTest, DistribDocExpirationUpdateProcessorTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=ACA3BF961C3082C4 -Dtests.slow=true -Dtests.locale=de_CH -Dtests.timezone=America/Shiprock -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ACA3BF961C3082C4]:0)
   [junit4] Completed in 7224.98s, 1 test, 2 errors <<< FAILURES!

[...truncated 697 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:519: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:61: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:187: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:510: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1349: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:956: There were test failures: 460 suites, 1857 tests, 1 suite-level error, 1 error, 903 ignored (15 assumptions)

Total time: 211 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC (asserts: true)
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message