lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build # 8760 - Failure!
Date Fri, 20 Dec 2013 15:19:31 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8760/
Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=937, name=coreLoadExecutor-682-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=937, name=coreLoadExecutor-682-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
	at __randomizedtesting.SeedInfo.seed([7BF83EF88B5E8D5F]:0)
	at java.lang.Thread.getStackTrace(Thread.java:1588)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:551)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1041)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:933)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:925)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:254)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:468)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:759)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:628)
	at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:229)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:580)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@219409bd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5c817380)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	... 1 more




Build Log:
[...truncated 10279 lines...]
   [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
   [junit4]   2> 268412 T857 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 268415 T857 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1387551517744
   [junit4]   2> 268416 T857 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 268416 T858 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 268516 T857 oasc.ZkTestServer.run start zk server on port:34054
   [junit4]   2> 268517 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 268627 T864 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3adcceb6 name:ZooKeeperConnection Watcher:127.0.0.1:34054 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 268627 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 268627 T857 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 268632 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 268633 T866 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c517fac name:ZooKeeperConnection Watcher:127.0.0.1:34054/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 268634 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 268634 T857 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 268636 T857 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 268639 T857 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 268641 T857 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 268643 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 268643 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 268653 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 268653 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 268658 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 268658 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 268661 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 268661 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 268664 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 268664 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 268667 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 268667 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 268669 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 268670 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 268672 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 268673 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 268675 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 268676 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 268678 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 268678 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 268680 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 268681 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 268751 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 268753 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46365
   [junit4]   2> 268754 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 268754 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 268755 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011
   [junit4]   2> 268755 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/'
   [junit4]   2> 268768 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/solr.xml
   [junit4]   2> 268830 T857 oasc.CoreContainer.<init> New CoreContainer 26414890
   [junit4]   2> 268831 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/]
   [junit4]   2> 268832 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 268832 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 268832 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 268832 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 268833 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 268833 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 268833 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 268833 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 268833 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 268837 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 268838 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 268838 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 268838 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34054/solr
   [junit4]   2> 268838 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 268839 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 268841 T877 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c6575ef name:ZooKeeperConnection Watcher:127.0.0.1:34054 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 268841 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 268844 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 268845 T879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b82baca name:ZooKeeperConnection Watcher:127.0.0.1:34054/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 268845 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 268847 T857 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 268850 T857 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 268852 T857 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 268853 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46365_
   [junit4]   2> 268854 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46365_
   [junit4]   2> 268857 T857 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 268863 T857 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 268865 T857 oasc.Overseer.start Overseer (id=90934576267001859-127.0.0.1:46365_-n_0000000000) starting
   [junit4]   2> 268868 T857 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 268871 T881 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 268871 T857 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 268873 T857 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 268874 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 268876 T880 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 268879 T882 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 268880 T882 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 268882 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 268882 T882 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 268883 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 268884 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46365",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46365_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 268884 T880 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 268884 T880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 268887 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 268887 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 269882 T882 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 269882 T882 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/collection1
   [junit4]   2> 269883 T882 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 269883 T882 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 269884 T882 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 269885 T882 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/collection1/'
   [junit4]   2> 269886 T882 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/collection1/lib/classes/' to classloader
   [junit4]   2> 269886 T882 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/collection1/lib/README' to classloader
   [junit4]   2> 269932 T882 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 269981 T882 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 269982 T882 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 269992 T882 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 270310 T882 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 270312 T882 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 270313 T882 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 270317 T882 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 270340 T882 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 270341 T882 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1387551518011/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1387551517744/control/data/
   [junit4]   2> 270341 T882 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7060b63
   [junit4]   2> 270341 T882 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1387551517744/control/data
   [junit4]   2> 270342 T882 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1387551517744/control/data/index/
   [junit4]   2> 270342 T882 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1387551517744/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 270342 T882 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1387551517744/control/data/index
   [junit4]   2> 270343 T882 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=47.005859375, floorSegmentMB=1.0087890625, forceMergeDeletesPctAllowed=9.307779485474738, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 270343 T882 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@12625bb6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@77731327),segFN=segments_1,generation=1}
   [junit4]   2> 270344 T882 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 270346 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 270347 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 270347 T882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 270347 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 270347 T882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 270347 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 270348 T882 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 270348 T882 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 270348 T882 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 270349 T882 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 270349 T882 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 270349 T882 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 270349 T882 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 270350 T882 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 270350 T882 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 270350 T882 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 270351 T882 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 270353 T882 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 270355 T882 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 270355 T882 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 270356 T882 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 270356 T882 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@12625bb6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@77731327),segFN=segments_1,generation=1}
   [junit4]   2> 270357 T882 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 270357 T882 oass.SolrIndexSearcher.<init> Opening Searcher@24886eba main
   [junit4]   2> 270359 T883 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24886eba main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 270360 T882 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 270360 T882 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46365 collection:control_collection shard:shard1
   [junit4]   2> 270361 T882 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 270368 T882 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 270370 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270371 T882 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 270371 T882 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C450 name=collection1 org.apache.solr.core.SolrCore@76918c6c url=http://127.0.0.1:46365/collection1 node=127.0.0.1:46365_ C450_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:46365, core=collection1, node_name=127.0.0.1:46365_}
   [junit4]   2> 270371 T882 C450 P46365 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46365/collection1/
   [junit4]   2> 270371 T882 C450 P46365 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 270372 T882 C450 P46365 oasc.SyncStrategy.syncToMe http://127.0.0.1:46365/collection1/ has no replicas
   [junit4]   2> 270372 T882 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46365/collection1/ shard1
   [junit4]   2> 270372 T882 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 270372 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 270375 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270378 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270381 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270484 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 270529 T882 oasc.ZkController.register We are http://127.0.0.1:46365/collection1/ and leader is http://127.0.0.1:46365/collection1/
   [junit4]   2> 270530 T882 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46365
   [junit4]   2> 270530 T882 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 270530 T882 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 270530 T882 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 270531 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270531 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270532 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270532 T882 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 270533 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 270533 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 270533 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 270534 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46365",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46365_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 270535 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 270536 T886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b92428e name:ZooKeeperConnection Watcher:127.0.0.1:34054/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 270536 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 270537 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 270538 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 270540 T857 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 270619 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 270621 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39202
   [junit4]   2> 270622 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 270623 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 270623 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869
   [junit4]   2> 270623 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/'
   [junit4]   2> 270637 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/solr.xml
   [junit4]   2> 270640 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 270640 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 270676 T857 oasc.CoreContainer.<init> New CoreContainer 1730537582
   [junit4]   2> 270676 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/]
   [junit4]   2> 270677 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 270677 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 270678 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 270678 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 270678 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 270678 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 270679 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 270679 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 270679 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 270683 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 270683 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 270683 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 270684 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34054/solr
   [junit4]   2> 270684 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 270685 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 270686 T897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6668ddba name:ZooKeeperConnection Watcher:127.0.0.1:34054 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 270687 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 270689 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 270691 T899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2412aef2 name:ZooKeeperConnection Watcher:127.0.0.1:34054/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 270691 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 270694 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 271697 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39202_
   [junit4]   2> 271698 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39202_
   [junit4]   2> 271702 T886 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 271702 T899 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 271702 T879 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 271708 T900 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 271708 T900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 271709 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271709 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271709 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271709 T900 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 271710 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 271711 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39202",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39202_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 271712 T880 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 271712 T880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 271714 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 271818 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 271818 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 271818 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 272710 T900 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 272710 T900 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/collection1
   [junit4]   2> 272710 T900 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 272711 T900 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 272711 T900 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 272713 T900 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/collection1/'
   [junit4]   2> 272714 T900 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/collection1/lib/classes/' to classloader
   [junit4]   2> 272714 T900 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/collection1/lib/README' to classloader
   [junit4]   2> 272755 T900 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 272788 T900 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 272790 T900 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 272796 T900 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 273133 T900 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 273135 T900 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 273137 T900 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 273140 T900 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 273163 T900 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 273163 T900 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1387551519869/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty1/
   [junit4]   2> 273163 T900 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7060b63
   [junit4]   2> 273164 T900 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty1
   [junit4]   2> 273164 T900 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty1/index/
   [junit4]   2> 273164 T900 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 273165 T900 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty1/index
   [junit4]   2> 273165 T900 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=47.005859375, floorSegmentMB=1.0087890625, forceMergeDeletesPctAllowed=9.307779485474738, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 273165 T900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@14e38aca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66acde7c),segFN=segments_1,generation=1}
   [junit4]   2> 273166 T900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 273169 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 273169 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 273170 T900 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 273170 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 273170 T900 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 273170 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 273170 T900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 273171 T900 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 273171 T900 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 273171 T900 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 273172 T900 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 273172 T900 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 273172 T900 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 273172 T900 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 273173 T900 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 273173 T900 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 273174 T900 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 273176 T900 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 273178 T900 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 273178 T900 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 273179 T900 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 273179 T900 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@14e38aca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66acde7c),segFN=segments_1,generation=1}
   [junit4]   2> 273180 T900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 273180 T900 oass.SolrIndexSearcher.<init> Opening Searcher@716d720c main
   [junit4]   2> 273183 T901 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@716d720c main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 273185 T900 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 273185 T900 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39202 collection:collection1 shard:shard1
   [junit4]   2> 273186 T900 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 273192 T900 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 273193 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273194 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273194 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273194 T900 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 273194 T900 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C451 name=collection1 org.apache.solr.core.SolrCore@78624ee7 url=http://127.0.0.1:39202/collection1 node=127.0.0.1:39202_ C451_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:39202, core=collection1, node_name=127.0.0.1:39202_}
   [junit4]   2> 273194 T900 C451 P39202 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39202/collection1/
   [junit4]   2> 273194 T900 C451 P39202 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 273195 T900 C451 P39202 oasc.SyncStrategy.syncToMe http://127.0.0.1:39202/collection1/ has no replicas
   [junit4]   2> 273195 T900 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39202/collection1/ shard1
   [junit4]   2> 273195 T900 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 273195 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 273198 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273199 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273202 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273305 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 273305 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 273305 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 273350 T900 oasc.ZkController.register We are http://127.0.0.1:39202/collection1/ and leader is http://127.0.0.1:39202/collection1/
   [junit4]   2> 273351 T900 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39202
   [junit4]   2> 273351 T900 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 273351 T900 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 273351 T900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 273408 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273408 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273408 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273408 T900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 273409 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 273409 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 273409 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 273410 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39202",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39202_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 273414 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 273483 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 273487 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40205
   [junit4]   2> 273487 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 273487 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 273488 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738
   [junit4]   2> 273488 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/'
   [junit4]   2> 273499 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/solr.xml
   [junit4]   2> 273516 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 273516 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 273516 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 273534 T857 oasc.CoreContainer.<init> New CoreContainer 987831332
   [junit4]   2> 273535 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/]
   [junit4]   2> 273536 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 273536 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 273536 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 273536 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 273536 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 273537 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 273537 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 273537 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 273537 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 273541 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 273541 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 273541 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 273542 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34054/solr
   [junit4]   2> 273542 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 273543 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 273544 T913 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63b07bd1 name:ZooKeeperConnection Watcher:127.0.0.1:34054 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 273545 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 273547 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 273548 T915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@457ec4ab name:ZooKeeperConnection Watcher:127.0.0.1:34054/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 273548 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 273552 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 274554 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40205_
   [junit4]   2> 274556 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40205_
   [junit4]   2> 274558 T886 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 274558 T879 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 274559 T899 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 274559 T915 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 274564 T916 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 274564 T916 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 274565 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274565 T916 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 274565 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274565 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274567 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 274567 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40205",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40205_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 274568 T880 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 274568 T880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 274570 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 274716 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 274716 T915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 274716 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 274716 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 275565 T916 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 275566 T916 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/collection1
   [junit4]   2> 275566 T916 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 275567 T916 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 275568 T916 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 275570 T916 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/collection1/'
   [junit4]   2> 275571 T916 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/collection1/lib/classes/' to classloader
   [junit4]   2> 275571 T916 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/collection1/lib/README' to classloader
   [junit4]   2> 275624 T916 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 275654 T916 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 275656 T916 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 275662 T916 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 275971 T916 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 275974 T916 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 275975 T916 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 275979 T916 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 275997 T916 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 275997 T916 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1387551522738/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty2/
   [junit4]   2> 275997 T916 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7060b63
   [junit4]   2> 275998 T916 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty2
   [junit4]   2> 275998 T916 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty2/index/
   [junit4]   2> 275998 T916 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 275999 T916 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1387551517744/jetty2/index
   [junit4]   2> 275999 T916 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=47.005859375, floorSegmentMB=1.0087890625, forceMergeDeletesPctAllowed=9.307779485474738, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 275999 T916 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@197320c9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@556d792c),segFN=segments_1,generation=1}
   [junit4]   2> 276000 T916 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 276003 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 276003 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 276003 T916 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 276004 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 276004 T916 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 276004 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 276004 T916 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 276005 T916 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 276005 T916 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 276005 T916 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 276006 T916 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 276006 T916 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 276006 T916 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 276006 T916 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 276007 T916 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 276007 T916 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 276008 T916 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 276010 T916 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 276013 T916 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 276014 T916 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 276014 T916 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 276014 T916 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@197320c9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@556d792c),segFN=segments_1,generation=1}
   [junit4]   2> 276015 T916 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 276015 T916 oass.SolrIndexSearcher.<init> Opening Searcher@6fe152d9 main
   [junit4]   2> 276018 T917 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6fe152d9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 276020 T916 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 276020 T916 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40205 collection:collection1 shard:shard1
   [junit4]   2> 276022 T916 oasc.ZkController.register We are http://127.0.0.1:40205/collection1/ and leader is http://127.0.0.1:39202/collection1/
   [junit4]   2> 276023 T916 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40205
   [junit4]   2> 276023 T916 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 276023 T916 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C452 name=collection1 org.apache.solr.core.SolrCore@1666a145 url=http://127.0.0.1:40205/collection1 node=127.0.0.1:40205_ C452_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40205, core=collection1, node_name=127.0.0.1:40205_}
   [junit4]   2> 276023 T918 C452 P40205 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 276024 T918 C452 P40205 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 276024 T916 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 276024 T918 C452 P40205 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 276024 T918 C452 P40205 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 276025 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 276025 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 276025 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 276025 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 276026 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 276027 T857 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 276027 T892 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 276027 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 276029 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40205",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40205_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 276032 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C453 name=collection1 org.apache.solr.core.SolrCore@76918c6c url=http://127.0.0.1:46365/collection1 node=127.0.0.1:46365_ C453_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:46365, core=collection1, node_name=127.0.0.1:46365_, leader=true}
   [junit4]   2> 276039 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-0 (1454953228261130241)]} 0 3
   [junit4]   2> 276039 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-0 (1454953228261130240)]} 0 3
   [junit4]   2> ASYNC  NEW_CORE C454 name=collection1 org.apache.solr.core.SolrCore@78624ee7 url=http://127.0.0.1:39202/collection1 node=127.0.0.1:39202_ C454_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:39202, core=collection1, node_name=127.0.0.1:39202_, leader=true}
   [junit4]   2> 276047 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-0 (1454953228270567424)]} 0 2
   [junit4]   2> 276048 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-0 (1454953228270567425)]} 0 2
   [junit4]   2> 276067 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-1 (1454953228291538945)]} 0 1
   [junit4]   2> 276068 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-1 (1454953228291538944)]} 0 2
   [junit4]   2> 276071 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-1 (1454953228295733248)]} 0 2
   [junit4]   2> 276071 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-1 (1454953228295733249)]} 0 1
   [junit4]   2> 276134 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276134 T915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276134 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276134 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276147 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&CONTROL=TRUE&version=2} {delete=[2-0 (-1454953228375425024)]} 0 1
   [junit4]   2> 276147 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&CONTROL=TRUE&version=2} {delete=[1-0 (-1454953228375425025)]} 0 1
   [junit4]   2>  C452_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=http://127.0.0.1:40205, core=collection1, node_name=127.0.0.1:40205_}
   [junit4]   2> 276157 T910 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {delete=[2-0 (-1454953228379619329)]} 0 1
   [junit4]   2> 276158 T911 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {delete=[1-0 (-1454953228379619328)]} 0 0
   [junit4]   2> 276158 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2-0 (-1454953228379619329)]} 0 8
   [junit4]   2> 276158 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1-0 (-1454953228379619328)]} 0 8
   [junit4]   2> 276161 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-2 (1454953228391153664)]} 0 1
   [junit4]   2> 276162 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-2 (1454953228391153665)]} 0 1
   [junit4]   2> 276168 T908 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {add=[2-2 (1454953228394299392)]} 0 1
   [junit4]   2> 276169 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-2 (1454953228394299392)]} 0 5
   [junit4]   2> 276169 T909 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {add=[1-2 (1454953228394299393)]} 0 2
   [junit4]   2> 276169 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-2 (1454953228394299393)]} 0 5
   [junit4]   2> 276201 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-3 (1454953228432048128)]} 0 1
   [junit4]   2> 276202 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-3 (1454953228433096704)]} 0 1
   [junit4]   2> 276214 T911 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {add=[2-3 (1454953228435193856)]} 0 1
   [junit4]   2> 276215 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-3 (1454953228435193856)]} 0 12
   [junit4]   2> 276215 T910 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {add=[1-3 (1454953228436242432)]} 0 2
   [junit4]   2> 276216 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-3 (1454953228436242432)]} 0 12
   [junit4]   2> 276228 T857 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 40205
   [junit4]   2> 276229 T857 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=987831332
   [junit4]   2> 276230 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 276230 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 276230 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 276232 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 276233 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40205",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:40205_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 276233 T880 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 276235 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 276287 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&CONTROL=TRUE&version=2} {delete=[2-3 (-1454953228523274240)]} 0 1
   [junit4]   2> 276287 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&CONTROL=TRUE&version=2} {delete=[1-3 (-1454953228523274241)]} 0 0
   [junit4]   2> 276292 T908 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {delete=[2-3 (-1454953228526419969)]} 0 0
   [junit4]   2> 276292 T909 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {delete=[1-3 (-1454953228526419968)]} 0 0
   [junit4]   2> 276293 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2-3 (-1454953228526419969)]} 0 3
   [junit4]   2> 276293 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1-3 (-1454953228526419968)]} 0 3
   [junit4]   2> 276296 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-4 (1454953228531662848)]} 0 1
   [junit4]   2> 276296 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-4 (1454953228531662849)]} 0 1
   [junit4]   2> 276301 T911 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {add=[2-4 (1454953228534808577)]} 0 1
   [junit4]   2> 276301 T910 C452 P40205 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2&distrib.from=http://127.0.0.1:39202/collection1/} {add=[1-4 (1454953228534808576)]} 0 1
   [junit4]   2> 276301 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-4 (1454953228534808577)]} 0 3
   [junit4]   2> 276302 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-4 (1454953228534808576)]} 0 4
   [junit4]   2> 276338 T899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276338 T886 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276338 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276338 T915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 276390 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-5 (1454953228630228992)]} 0 1
   [junit4]   2> 276391 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-5 (1454953228630228993)]} 0 2
   [junit4]   2> 276394 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-5 (1454953228634423296)]} 0 2
   [junit4]   2> 276394 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-5 (1454953228634423297)]} 0 1
   [junit4]   2> 276444 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-6 (1454953228687900672)]} 0 0
   [junit4]   2> 276445 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-6 (1454953228688949248)]} 0 0
   [junit4]   2> 276448 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-6 (1454953228691046400)]} 0 1
   [junit4]   2> 276448 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-6 (1454953228692094976)]} 0 0
   [junit4]   2> 276526 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-7 (1454953228773883904)]} 0 1
   [junit4]   2> 276527 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-7 (1454953228773883905)]} 0 2
   [junit4]   2> 276529 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-7 (1454953228777029632)]} 0 0
   [junit4]   2> 276530 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-7 (1454953228777029633)]} 0 1
   [junit4]   2> 276583 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-8 (1454953228832604160)]} 0 1
   [junit4]   2> 276583 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-8 (1454953228832604161)]} 0 1
   [junit4]   2> 276586 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-8 (1454953228836798464)]} 0 1
   [junit4]   2> 276587 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-8 (1454953228836798465)]} 0 2
   [junit4]   2> 276623 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&CONTROL=TRUE&version=2} {delete=[1-4 (-1454953228875595776)]} 0 0
   [junit4]   2> 276623 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&CONTROL=TRUE&version=2} {delete=[2-4 (-1454953228875595777)]} 0 0
   [junit4]   2> 276626 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2-4 (-1454953228878741504)]} 0 0
   [junit4]   2> 276626 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1-4 (-1454953228878741505)]} 0 0
   [junit4]   2> 276629 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-9 (1454953228881887232)]} 0 1
   [junit4]   2> 276629 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-9 (1454953228881887233)]} 0 1
   [junit4]   2> 276633 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-9 (1454953228885032961)]} 0 1
   [junit4]   2> 276633 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-9 (1454953228885032960)]} 0 1
   [junit4]   2> 276663 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-10 (1454953228917538816)]} 0 1
   [junit4]   2> 276664 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-10 (1454953228917538817)]} 0 2
   [junit4]   2> 276666 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-10 (1454953228920684544)]} 0 0
   [junit4]   2> 276669 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-10 (1454953228920684545)]} 0 3
   [junit4]   2> 276682 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-11 (1454953228936413184)]} 0 1
   [junit4]   2> 276684 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-11 (1454953228938510336)]} 0 1
   [junit4]   2> 276686 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-11 (1454953228940607488)]} 0 1
   [junit4]   2> 276687 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-11 (1454953228941656064)]} 0 1
   [junit4]   2> 276745 T875 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-12 (1454953229002473472)]} 0 1
   [junit4]   2> 276746 T874 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-12 (1454953229003522048)]} 0 1
   [junit4]   2> 276749 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-12 (1454953229006667776)]} 0 1
   [junit4]   2> 276750 T895 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-12 (1454953229007716352)]} 0 1
   [junit4]   2> 276775 T872 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-13 (1454953229033930752)]} 0 2
   [junit4]   2> 276775 T873 C453 P46365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-13 (1454953229033930753)]} 0 1
   [junit4]   2> 276779 T894 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2-13 (1454953229038125056)]} 0 1
   [junit4]   2> 276780 T893 C454 P39202 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1-13 (1454953229038125057)]} 0 2
   [junit4]   2> 276857 T875 C453 P46365 oasup.LogUpdateProcessor.

[...truncated too long message...]

it4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:201)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:198)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:198)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:409)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:378)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:365)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:102)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:277)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 421768 T936 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/90934576267001866-core_node2-n_0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:169)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:166)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:166)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:66)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:280)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 421768 T936 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 421768 T936 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 421768 T936 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 421768 T936 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 421768 T936 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=en_US, timezone=Arctic/Longyearbyen
   [junit4]   2> NOTE: Linux 3.8.0-34-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=1,free=143360040,total=531628032
   [junit4]   2> NOTE: All tests run in this JVM: [TestDynamicFieldCollectionResource, TestMergePolicyConfig, UpdateParamsTest, JsonLoaderTest, TestCollapseQParserPlugin, TestCloudManagedSchemaAddField, TestFaceting, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, SuggesterTest, BasicDistributedZkTest, TestReloadAndDeleteDocs, TestSolrXMLSerializer, DateMathParserTest, NoFacetTest, TestSuggestSpellingConverter, TestSolrQueryParser, TestSearchPerf, QueryEqualityTest, ResourceLoaderTest, TestPHPSerializedResponseWriter, SolrIndexSplitterTest, ModifyConfFileTest, ResponseLogComponentTest, DirectUpdateHandlerOptimizeTest, SliceStateUpdateTest, TestDocBasedVersionConstraints, TestConfig, TestSchemaVersionResource, CircularListTest, QueryResultKeyTest, BadComponentTest, CopyFieldTest, TestCoreContainer, TestReversedWildcardFilterFactory, TestSolrDeletionPolicy1, SuggesterFSTTest, RecoveryZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryZkTest -Dtests.seed=7BF83EF88B5E8D5F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_US -Dtests.timezone=Arctic/Longyearbyen -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | RecoveryZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=937, name=coreLoadExecutor-682-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7BF83EF88B5E8D5F]:0)
   [junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1588)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:551)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1041)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:933)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:925)
   [junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
   [junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:254)
   [junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:468)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:759)
   [junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:628)
   [junit4]    > 	at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:229)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:580)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:258)
   [junit4]    > 	at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:250)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]    > 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@219409bd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5c817380)
   [junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
   [junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
   [junit4]    > 	... 1 more
   [junit4] Completed on J0 in 153.66s, 1 test, 1 error <<< FAILURES!

[...truncated 849 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1304: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:937: There were test failures: 357 suites, 1572 tests, 1 suite-level error, 34 ignored (6 assumptions)

Total time: 50 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message