lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1165 - Still Failing!
Date Sun, 29 Dec 2013 00:01:50 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1165/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.DeleteInactiveReplicaTest:     1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.DeleteInactiveReplicaTest: 
   1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([CAFADD75A6A985CD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DeleteInactiveReplicaTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:503)         at java.util.TimerThread.mainLoop(Timer.java:526)         at java.util.TimerThread.run(Timer.java:505)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        at java.util.TimerThread.run(Timer.java:505)
	at __randomizedtesting.SeedInfo.seed([CAFADD75A6A985CD]:0)




Build Log:
[...truncated 10262 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteInactiveReplicaTest
   [junit4]   2> 1046490 T4166 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1046497 T4166 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-DeleteInactiveReplicaTest-1388275184917
   [junit4]   2> 1046498 T4166 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1046500 T4167 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1046600 T4166 oasc.ZkTestServer.run start zk server on port:50279
   [junit4]   2> 1046603 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1046611 T4174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7afd9145 name:ZooKeeperConnection Watcher:127.0.0.1:50279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1046611 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1046612 T4166 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1046630 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1046634 T4177 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3af00665 name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1046634 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1046635 T4166 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1046650 T4166 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1046661 T4166 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1046674 T4166 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1046689 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1046690 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1046705 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1046706 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1046723 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1046724 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1046734 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1046735 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1046748 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1046750 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1046765 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1046766 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1046780 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1046782 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1046791 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1046792 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1046803 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1046805 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1046817 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1046818 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1046828 T4166 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1046829 T4166 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1047288 T4166 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1047299 T4166 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50282
   [junit4]   2> 1047301 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1047302 T4166 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1047302 T4166 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265
   [junit4]   2> 1047302 T4166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/'
   [junit4]   2> 1047354 T4166 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/solr.xml
   [junit4]   2> 1047457 T4166 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1047458 T4166 oasc.CoreContainer.<init> New CoreContainer 370497089
   [junit4]   2> 1047459 T4166 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/]
   [junit4]   2> 1047462 T4166 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1047462 T4166 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1047462 T4166 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1047463 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1047463 T4166 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1047463 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1047464 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1047464 T4166 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1047465 T4166 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1047481 T4166 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1047481 T4166 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1047482 T4166 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1047482 T4166 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50279/solr
   [junit4]   2> 1047482 T4166 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1047486 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1047493 T4189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@127a9665 name:ZooKeeperConnection Watcher:127.0.0.1:50279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1047493 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1047501 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1047506 T4192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@563275e2 name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1047507 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1047513 T4166 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1047526 T4166 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1047541 T4166 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1047549 T4166 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50282_
   [junit4]   2> 1047554 T4166 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50282_
   [junit4]   2> 1047570 T4166 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1047595 T4166 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1047607 T4166 oasc.Overseer.start Overseer (id=90982002519244803-127.0.0.1:50282_-n_0000000000) starting
   [junit4]   2> 1047623 T4166 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1047637 T4194 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1047639 T4166 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1047648 T4166 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1047654 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1047672 T4193 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1047679 T4166 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265
   [junit4]   2> 1047823 T4166 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/conf/
   [junit4]   2> 1047829 T4166 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1047830 T4166 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1047831 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1047839 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1047845 T4198 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e0c9232 name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1047845 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1047850 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1047859 T4166 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1048239 T4166 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1048244 T4166 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50286
   [junit4]   2> 1048245 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1048246 T4166 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1048246 T4166 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279
   [junit4]   2> 1048246 T4166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/'
   [junit4]   2> 1048287 T4166 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/solr.xml
   [junit4]   2> 1048381 T4166 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1048382 T4166 oasc.CoreContainer.<init> New CoreContainer 1598892741
   [junit4]   2> 1048382 T4166 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/]
   [junit4]   2> 1048385 T4166 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1048385 T4166 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1048385 T4166 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1048386 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1048386 T4166 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1048387 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1048387 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1048388 T4166 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1048388 T4166 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1048408 T4166 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1048409 T4166 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1048409 T4166 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1048410 T4166 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50279/solr
   [junit4]   2> 1048411 T4166 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1048414 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048425 T4210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c3127c5 name:ZooKeeperConnection Watcher:127.0.0.1:50279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048426 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048435 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1048448 T4213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d35ec5 name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1048449 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1048467 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1049478 T4166 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50286_
   [junit4]   2> 1049482 T4166 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50286_
   [junit4]   2> 1049492 T4192 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1049492 T4198 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1049493 T4213 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1049505 T4166 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279
   [junit4]   2> 1049619 T4166 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/conf/
   [junit4]   2> 1049625 T4166 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1049626 T4166 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1049626 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1049957 T4166 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1049963 T4166 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50289
   [junit4]   2> 1049965 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1049966 T4166 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1049966 T4166 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050
   [junit4]   2> 1049966 T4166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/'
   [junit4]   2> 1050018 T4166 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/solr.xml
   [junit4]   2> 1050133 T4166 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1050134 T4166 oasc.CoreContainer.<init> New CoreContainer 489880285
   [junit4]   2> 1050135 T4166 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/]
   [junit4]   2> 1050138 T4166 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1050138 T4166 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1050138 T4166 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1050139 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1050140 T4166 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1050140 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1050140 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1050141 T4166 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1050142 T4166 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1050157 T4166 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1050159 T4166 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1050159 T4166 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1050159 T4166 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50279/solr
   [junit4]   2> 1050159 T4166 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1050161 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1050170 T4226 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d06e5d4 name:ZooKeeperConnection Watcher:127.0.0.1:50279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1050171 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1050180 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1050186 T4229 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56798ddc name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1050187 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1050201 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1051214 T4166 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50289_
   [junit4]   2> 1051218 T4166 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50289_
   [junit4]   2> 1051227 T4213 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1051228 T4192 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1051228 T4198 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1051229 T4229 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1051244 T4166 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050
   [junit4]   2> 1051433 T4166 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/conf/
   [junit4]   2> 1051442 T4166 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1051443 T4166 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1051443 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1051857 T4166 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1051860 T4166 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50292
   [junit4]   2> 1051863 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1051864 T4166 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1051864 T4166 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866
   [junit4]   2> 1051864 T4166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/'
   [junit4]   2> 1051904 T4166 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/solr.xml
   [junit4]   2> 1051980 T4166 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1051980 T4166 oasc.CoreContainer.<init> New CoreContainer 2047346977
   [junit4]   2> 1051981 T4166 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/]
   [junit4]   2> 1051982 T4166 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1051982 T4166 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1051983 T4166 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1051983 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1051983 T4166 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1051984 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1051984 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1051984 T4166 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1051985 T4166 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1051999 T4166 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1051999 T4166 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1052000 T4166 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1052000 T4166 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50279/solr
   [junit4]   2> 1052001 T4166 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1052004 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1052008 T4242 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b8b86bc name:ZooKeeperConnection Watcher:127.0.0.1:50279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1052008 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1052015 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1052021 T4245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45417980 name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1052022 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1052034 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1053044 T4166 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50292_
   [junit4]   2> 1053047 T4166 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50292_
   [junit4]   2> 1053059 T4192 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1053059 T4245 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1053059 T4229 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1053060 T4198 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1053060 T4213 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1053076 T4166 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866
   [junit4]   2> 1053266 T4166 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/conf/
   [junit4]   2> 1053280 T4166 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1053281 T4166 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1053281 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1053686 T4166 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1053691 T4166 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50295
   [junit4]   2> 1053693 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1053694 T4166 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1053694 T4166 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty4-1388275191704
   [junit4]   2> 1053695 T4166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty4-1388275191704/'
   [junit4]   2> 1053730 T4166 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty4-1388275191704/solr.xml
   [junit4]   2> 1053803 T4166 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 1053803 T4166 oasc.CoreContainer.<init> New CoreContainer 1053259982
   [junit4]   2> 1053804 T4166 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty4-1388275191704/]
   [junit4]   2> 1053805 T4166 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1053806 T4166 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1053806 T4166 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1053806 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1053806 T4166 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1053807 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1053807 T4166 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1053807 T4166 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1053808 T4166 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1053821 T4166 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1053822 T4166 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1053822 T4166 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1053823 T4166 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50279/solr
   [junit4]   2> 1053824 T4166 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1053826 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1053831 T4258 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5348cc04 name:ZooKeeperConnection Watcher:127.0.0.1:50279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1053831 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1053838 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1053843 T4261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c7dd2f2 name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1053843 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1053856 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1054867 T4166 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50295_
   [junit4]   2> 1054871 T4166 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50295_
   [junit4]   2> 1054882 T4261 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1054883 T4213 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1054883 T4192 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1054883 T4229 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1054884 T4198 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1054885 T4245 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1054896 T4166 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty4-1388275191704
   [junit4]   2> 1055089 T4166 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty4-1388275191704/conf/
   [junit4]   2> 1055100 T4166 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1055101 T4166 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1055102 T4166 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1055110 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055115 T4265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d81f955 name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055116 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1055120 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1055132 T4166 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055136 T4268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57b6c95a name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055137 T4166 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1055142 T4166 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1055156 T4235 oasha.CollectionsHandler.handleCreateAction Creating Collection : action=CREATE&wt=javabin&maxShardsPerNode=2&version=2&replicationFactor=2&numShards=2&name=delDeadColl
   [junit4]   2> 1055165 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055178 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055179 T4194 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delDeadColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1055179 T4194 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delDeadColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 1055182 T4194 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 1055182 T4194 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delDeadColl 
   [junit4]   2> 1055182 T4194 oascc.SolrZkClient.makePath makePath: /collections/delDeadColl
   [junit4]   2> 1055213 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055228 T4193 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1055231 T4193 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: delDeadColl
   [junit4]   2> 1055231 T4193 oasc.Overseer$ClusterStateUpdater.createCollection Create collection delDeadColl with shards [shard1, shard2]
   [junit4]   2> 1055240 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055243 T4213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055244 T4229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055244 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055245 T4198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055245 T4261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055244 T4265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055244 T4245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055244 T4268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055315 T4194 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 1055315 T4194 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard1_replica1 as part of slice shard1 of collection delDeadColl on 127.0.0.1:50289_
   [junit4]   2> 1055316 T4194 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard1_replica2 as part of slice shard1 of collection delDeadColl on 127.0.0.1:50292_
   [junit4]   2> 1055317 T4194 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard2_replica1 as part of slice shard2 of collection delDeadColl on 127.0.0.1:50286_
   [junit4]   2> 1055321 T4194 oasc.OverseerCollectionProcessor.createCollection Creating shard delDeadColl_shard2_replica2 as part of slice shard2 of collection delDeadColl on 127.0.0.1:50282_
   [junit4]   2> 1055336 T4236 oasc.ZkController.publish publishing core=delDeadColl_shard1_replica2 state=down
   [junit4]   2> 1055337 T4207 oasc.ZkController.publish publishing core=delDeadColl_shard2_replica1 state=down
   [junit4]   2> 1055339 T4218 oasc.ZkController.publish publishing core=delDeadColl_shard1_replica1 state=down
   [junit4]   2> 1055342 T4182 oasc.ZkController.publish publishing core=delDeadColl_shard2_replica2 state=down
   [junit4]   2> 1055343 T4207 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055344 T4236 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055344 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055344 T4218 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055347 T4182 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1055350 T4193 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1055352 T4193 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50286",
   [junit4]   2> 	  "core":"delDeadColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50286_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055361 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055370 T4193 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50292",
   [junit4]   2> 	  "core":"delDeadColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50292_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055381 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055384 T4193 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50289",
   [junit4]   2> 	  "core":"delDeadColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50289_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055397 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055403 T4193 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50282",
   [junit4]   2> 	  "core":"delDeadColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50282_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"delDeadColl",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1055419 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1055530 T4265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055531 T4268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055531 T4192 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055531 T4198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055532 T4245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055531 T4229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055551 T4261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1055551 T4213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1056346 T4207 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard2_replica1
   [junit4]   2> 1056346 T4236 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard1_replica2
   [junit4]   2> 1056348 T4207 oasc.CoreContainer.create Creating SolrCore 'delDeadColl_shard2_replica1' using instanceDir: ./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1
   [junit4]   2> 1056349 T4207 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1056349 T4218 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard1_replica1
   [junit4]   2> 1056350 T4236 oasc.CoreContainer.create Creating SolrCore 'delDeadColl_shard1_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2
   [junit4]   2> 1056350 T4236 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1056351 T4182 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delDeadColl_shard2_replica2
   [junit4]   2> 1056351 T4218 oasc.CoreContainer.create Creating SolrCore 'delDeadColl_shard1_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1
   [junit4]   2> 1056352 T4218 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1056353 T4182 oasc.CoreContainer.create Creating SolrCore 'delDeadColl_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2
   [junit4]   2> 1056353 T4182 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delDeadColl
   [junit4]   2> 1056354 T4236 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056354 T4236 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1056355 T4182 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056355 T4182 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1056354 T4207 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056357 T4207 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1056354 T4218 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1056357 T4218 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delDeadColl
   [junit4]   2> 1056363 T4218 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1/'
   [junit4]   2> 1056364 T4207 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1/'
   [junit4]   2> 1056364 T4236 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2/'
   [junit4]   2> 1056364 T4182 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2/'
   [junit4]   2> 1056632 T4182 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056651 T4236 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056655 T4218 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056670 T4207 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1056849 T4182 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056906 T4218 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056907 T4182 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056909 T4236 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056914 T4236 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056920 T4218 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056926 T4207 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056939 T4182 oass.IndexSchema.readSchema [delDeadColl_shard2_replica2] Schema name=test
   [junit4]   2> 1056944 T4236 oass.IndexSchema.readSchema [delDeadColl_shard1_replica2] Schema name=test
   [junit4]   2> 1056951 T4207 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1056958 T4218 oass.IndexSchema.readSchema [delDeadColl_shard1_replica1] Schema name=test
   [junit4]   2> 1056987 T4207 oass.IndexSchema.readSchema [delDeadColl_shard2_replica1] Schema name=test
   [junit4]   2> 1058623 T4182 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1058642 T4182 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1058649 T4182 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1058666 T4218 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1058677 T4218 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1058691 T4218 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1058704 T4207 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1058726 T4207 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1058730 T4207 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1058762 T4182 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058784 T4182 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058788 T4236 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1058798 T4218 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058803 T4182 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1058807 T4207 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058809 T4182 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1058810 T4182 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1058810 T4182 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1058814 T4218 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058814 T4236 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1058817 T4182 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1058818 T4182 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1058818 T4182 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1058819 T4182 oasc.SolrCore.<init> [delDeadColl_shard2_replica2] Opening new SolrCore at ./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2/, dataDir=./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2/data/
   [junit4]   2> 1058819 T4182 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55cbd9a8
   [junit4]   2> 1058821 T4182 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2/data
   [junit4]   2> 1058821 T4182 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2/data/index/
   [junit4]   2> 1058821 T4182 oasc.SolrCore.initIndex WARN [delDeadColl_shard2_replica2] Solr index directory './org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1058822 T4207 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058822 T4182 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteInactiveReplicaTest-controljetty-1388275185265/delDeadColl_shard2_replica2/data/index
   [junit4]   2> 1058823 T4182 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=27.8623046875, floorSegmentMB=1.283203125, forceMergeDeletesPctAllowed=24.115981491658935, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2726954704703973
   [junit4]   2> 1058828 T4207 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1058830 T4182 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a2e26d6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@49f8e1dc),segFN=segments_1,generation=1}
   [junit4]   2> 1058831 T4182 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1058833 T4218 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1058835 T4236 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1058840 T4182 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1058842 T4182 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1058843 T4182 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1058843 T4182 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1058842 T4218 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1058845 T4218 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1058845 T4218 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1058844 T4182 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1058848 T4182 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1058844 T4207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1058849 T4182 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1058849 T4207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1058850 T4207 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1058851 T4218 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1058852 T4218 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1058853 T4218 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1058854 T4218 oasc.SolrCore.<init> [delDeadColl_shard1_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1/data/
   [junit4]   2> 1058849 T4182 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1058855 T4182 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1058855 T4207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1058856 T4182 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1058856 T4207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1058857 T4182 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1058858 T4182 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1058859 T4182 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1058854 T4218 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55cbd9a8
   [junit4]   2> 1058860 T4182 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1058861 T4218 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1/data
   [junit4]   2> 1058862 T4218 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1/data/index/
   [junit4]   2> 1058862 T4218 oasc.SolrCore.initIndex WARN [delDeadColl_shard1_replica1] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1058857 T4207 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1058861 T4182 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1058863 T4207 oasc.SolrCore.<init> [delDeadColl_shard2_replica1] Opening new SolrCore at ./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1/, dataDir=./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1/data/
   [junit4]   2> 1058866 T4218 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty2-1388275188050/delDeadColl_shard1_replica1/data/index
   [junit4]   2> 1058864 T4182 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1058867 T4218 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=27.8623046875, floorSegmentMB=1.283203125, forceMergeDeletesPctAllowed=24.115981491658935, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2726954704703973
   [junit4]   2> 1058868 T4182 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1058866 T4207 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55cbd9a8
   [junit4]   2> 1058871 T4207 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1/data
   [junit4]   2> 1058871 T4207 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1/data/index/
   [junit4]   2> 1058872 T4207 oasc.SolrCore.initIndex WARN [delDeadColl_shard2_replica1] Solr index directory './org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1058868 T4218 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@14e7c6c9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4868e15f),segFN=segments_1,generation=1}
   [junit4]   2> 1058873 T4218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1058878 T4207 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty1-1388275186279/delDeadColl_shard2_replica1/data/index
   [junit4]   2> 1058879 T4207 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=27.8623046875, floorSegmentMB=1.283203125, forceMergeDeletesPctAllowed=24.115981491658935, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2726954704703973
   [junit4]   2> 1058880 T4218 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1058880 T4218 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1058881 T4207 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6ae26f48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@19efe78f),segFN=segments_1,generation=1}
   [junit4]   2> 1058883 T4207 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1058884 T4182 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1058891 T4182 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1058892 T4207 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1058893 T4218 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1058894 T4218 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1058894 T4207 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1058895 T4207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1058896 T4207 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1058896 T4207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1058897 T4207 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1058892 T4182 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1058899 T4207 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1058900 T4207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1058901 T4207 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1058902 T4236 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058894 T4218 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1058902 T4207 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1058900 T4182 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@25a48bf3
   [junit4]   2> 1058904 T4207 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1058905 T4207 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1058905 T4207 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1058906 T4207 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1058908 T4207 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1058910 T4207 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1058903 T4218 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1058912 T4207 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1058911 T4236 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1058904 T4182 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a2e26d6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@49f8e1dc),segFN=segments_1,generation=1}
   [junit4]   2> 1058913 T4182 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1058914 T4182 oass.SolrIndexSearcher.<init> Opening Searcher@530b4668 main
   [junit4]   2> 1058912 T4218 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1059681 T4218 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1059681 T4218 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1059682 T4218 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1059683 T4218 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1059683 T4218 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1059683 T4218 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1058915 T4207 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1059686 T4218 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1059691 T4218 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1059692 T4218 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1059693 T4218 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1059694 T4182 oasc.CoreContainer.registerCore registering core: delDeadColl_shard2_replica2
   [junit4]   2> 1059695 T4218 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1059697 T4273 oasc.SolrCore.registerSearcher [delDeadColl_shard2_replica2] Registered new searcher Searcher@530b4668 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1059701 T4207 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1059702 T4207 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1059703 T4207 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@175d87b0
   [junit4]   2> 1059704 T4182 oasc.ZkController.register Register replica - core:delDeadColl_shard2_replica2 address:http://127.0.0.1:50282 collection:delDeadColl shard:shard2
   [junit4]   2> 1059706 T4182 oascc.SolrZkClient.makePath makePath: /collections/delDeadColl/leader_elect/shard2/election
   [junit4]   2> 1059707 T4218 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1059708 T4218 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1059703 T4236 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1059710 T4207 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6ae26f48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@19efe78f),segFN=segments_1,generation=1}
   [junit4]   2> 1059711 T4207 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1059711 T4218 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@11dbf0d6
   [junit4]   2> 1059711 T4207 oass.SolrIndexSearcher.<init> Opening Searcher@59db9780 main
   [junit4]   2> 1059713 T4218 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@14e7c6c9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4868e15f),segFN=segments_1,generation=1}
   [junit4]   2> 1059713 T4218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1059715 T4218 oass.SolrIndexSearcher.<init> Opening Searcher@68e90a07 main
   [junit4]   2> 1059734 T4218 oasc.CoreContainer.registerCore registering core: delDeadColl_shard1_replica1
   [junit4]   2> 1059735 T4218 oasc.ZkController.register Register replica - core:delDeadColl_shard1_replica1 address:http://127.0.0.1:50289 collection:delDeadColl shard:shard1
   [junit4]   2> 1059735 T4207 oasc.CoreContainer.registerCore registering core: delDeadColl_shard2_replica1
   [junit4]   2> 1059736 T4207 oasc.ZkController.register Register replica - core:delDeadColl_shard2_replica1 address:http://127.0.0.1:50286 collection:delDeadColl shard:shard2
   [junit4]   2> 1059736 T4274 oasc.SolrCore.registerSearcher [delDeadColl_shard2_replica1] Registered new searcher Searcher@59db9780 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1059738 T4275 oasc.SolrCore.registerSearcher [delDeadColl_shard1_replica1] Registered new searcher Searcher@68e90a07 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1059739 T4236 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1059740 T4236 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1059741 T4236 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1059740 T4218 oascc.SolrZkClient.makePath makePath: /collections/delDeadColl/leader_elect/shard1/election
   [junit4]   2> 1059742 T4207 oascc.SolrZkClient.makePath makePath: /collections/delDeadColl/leader_elect/shard2/election
   [junit4]   2> 1059747 T4236 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1059748 T4236 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1059749 T4236 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1059749 T4236 oasc.SolrCore.<init> [delDeadColl_shard1_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2/data/
   [junit4]   2> 1059751 T4236 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55cbd9a8
   [junit4]   2> 1059753 T4236 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2/data
   [junit4]   2> 1059755 T4236 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2/data/index/
   [junit4]   2> 1059755 T4236 oasc.SolrCore.initIndex WARN [delDeadColl_shard1_replica2] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1059758 T4182 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1059758 T4236 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.DeleteInactiveReplicaTest-jetty3-1388275189866/delDeadColl_shard1_replica2/data/index
   [junit4]   2> 1059763 T4236 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=27.8623046875, floorSegmentMB=1.283203125, forceMergeDeletesPctAllowed=24.115981491658935, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2726954704703973
   [junit4]   2> 1059770 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059771 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059771 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059772 T4236 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2f38788d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2856086f),segFN=segments_1,generation=1}
   [junit4]   2> 1059774 T4236 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1059774 T4182 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1059774 T4182 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1701 name=delDeadColl_shard2_replica2 org.apache.solr.core.SolrCore@4a809e6a url=http://127.0.0.1:50282/delDeadColl_shard2_replica2 node=127.0.0.1:50282_ C1701_STATE=coll:delDeadColl core:delDeadColl_shard2_replica2 props:{state=down, base_url=http://127.0.0.1:50282, core=delDeadColl_shard2_replica2, node_name=127.0.0.1:50282_}
   [junit4]   2> 1059775 T4182 C1701 P50282 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50282/delDeadColl_shard2_replica2/
   [junit4]   2> 1059775 T4182 C1701 P50282 oasu.PeerSync.sync PeerSync: core=delDeadColl_shard2_replica2 url=http://127.0.0.1:50282 START replicas=[http://127.0.0.1:50286/delDeadColl_shard2_replica1/] nUpdates=100
   [junit4]   2> 1059778 T4182 C1701 P50282 oasu.PeerSync.sync PeerSync: core=delDeadColl_shard2_replica2 url=http://127.0.0.1:50282 DONE.  We have no versions.  sync failed.
   [junit4]   2> 1059788 T4182 C1701 P50282 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 1059789 T4182 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 1059789 T4182 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50282/delDeadColl_shard2_replica2/ shard2
   [junit4]   2> 1059789 T4182 oascc.SolrZkClient.makePath makePath: /collections/delDeadColl/leaders/shard2
   [junit4]   2> 1059787 T4236 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1059783 T4193 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1059793 T4236 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> ASYNC  NEW_CORE C1702 name=delDeadColl_shard2_replica1 org.apache.solr.core.SolrCore@6b0ba0c8 url=http://127.0.0.1:50286/delDeadColl_shard2_replica1 node=127.0.0.1:50286_ C1702_STATE=coll:delDeadColl core:delDeadColl_shard2_replica1 props:{state=down, base_url=http://127.0.0.1:50286, core=delDeadColl_shard2_replica1, node_name=127.0.0.1:50286_}
   [junit4]   2> 1059791 T4203 C1702 P50286 oasc.SolrCore.execute [delDeadColl_shard2_replica1] webapp= path=/get params={wt=javabin&version=2&distrib=false&qt=/get&getVersions=100} status=0 QTime=1 
   [junit4]   2> 1059796 T4236 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1059797 T4236 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1059797 T4236 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1059797 T4236 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1059799 T4236 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1059799 T4236 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1059800 T4236 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1059801 T4236 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1059803 T4236 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1059803 T4236 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1059804 T4236 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1059805 T4218 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1059806 T4236 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1059807 T4236 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1059808 T4192 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1059809 T4236 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1059812 T4236 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1059814 T4265 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1059814 T4198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1059813 T4213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1059813 T4229 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1059813 T4245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1059813 T4261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncC

[...truncated too long message...]

xception EndOfStreamException: Unable to read additional data from client sessionid 0x1433ba6351d0010, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 1072281 T4198 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1072284 T4268 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1072284 T4261 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1072293 T4166 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1072349 T4166 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50289
   [junit4]   2> 1072349 T4166 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50292
   [junit4]   2> 1072349 T4166 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50295
   [junit4]   2> 1072350 T4166 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1053259982
   [junit4]   2> 1072357 T4268 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1072358 T4265 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1072359 T4198 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1072371 T4166 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1072431 T4166 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50282
   [junit4]   2> 1072435 T4166 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1072439 T4166 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50279 50279
   [junit4]   2> 1072552 T4268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57b6c95a name:ZooKeeperConnection Watcher:127.0.0.1:50279/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 1072553 T4268 oascc.ConnectionManager.process zkClient has disconnected
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1072558 T4166 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 26082 T4165 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 5 leaked thread(s).
   [junit4]   2> 1073705 T4192 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1073859 T4261 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 46191 T4165 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.DeleteInactiveReplicaTest: 
   [junit4]   2> 	   1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> 46192 T4165 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]   2> 49203 T4165 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:505)
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=en_AU, timezone=Africa/Lome
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=2,threads=2,free=61278840,total=185188352
   [junit4]   2> NOTE: All tests run in this JVM: [ResourceLoaderTest, FieldFacetExtrasTest, UpdateParamsTest, PolyFieldTest, TestRTGBase, AssignTest, DateFieldTest, BJQParserTest, TestPartialUpdateDeduplication, PreAnalyzedUpdateProcessorTest, ShardRoutingTest, QueryFacetTest, MinimalSchemaTest, NotRequiredUniqueKeyTest, SolrXmlInZkTest, TestCoreDiscovery, SpellCheckComponentTest, TestCollapseQParserPlugin, TestLMJelinekMercerSimilarityFactory, TestSchemaNameResource, TestJmxMonitoredMap, TriLevelCompositeIdRoutingTest, CacheHeaderTest, TestReplicationHandler, OpenExchangeRatesOrgProviderTest, TestFieldCollectionResource, SpellingQueryConverterTest, SolrCmdDistributorTest, SolrIndexConfigTest, ResponseLogComponentTest, TestSolrQueryParser, TestBadConfig, TestPostingsSolrHighlighter, TermVectorComponentDistributedTest, OutputWriterTest, TestClassNameShortening, SortByFunctionTest, PrimitiveFieldTypeTest, CachingDirectoryFactoryTest, FastVectorHighlighterTest, HdfsLockFactoryTest, ReturnFieldsTest, HdfsRecoveryZkTest, TestSchemaVersionResource, TestDefaultSimilarityFactory, SliceStateUpdateTest, RequestHandlersTest, TestQuerySenderNoQuery, TestSolrQueryParserDefaultOperatorResource, HardAutoCommitTest, TestReloadAndDeleteDocs, TestManagedSchema, ZkSolrClientTest, BasicFunctionalityTest, TestUniqueKeyFieldResource, RAMDirectoryFactoryTest, TestRandomFaceting, TestSolrXmlPersistor, TestLazyCores, EnumFieldTest, XsltUpdateRequestHandlerTest, ClusterStateTest, OverseerCollectionProcessorTest, TestBinaryResponseWriter, TermsComponentTest, TestRemoteStreaming, TestJoin, InfoHandlerTest, TestAnalyzedSuggestions, ZkCLITest, TestHighFrequencyDictionaryFactory, TestSolrQueryParserResource, QueryEqualityTest, BlockCacheTest, DistributedSpellCheckComponentTest, LukeRequestHandlerTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, TestImplicitCoreProperties, SpellCheckCollatorTest, TestElisionMultitermQuery, RecoveryZkTest, TestDocSet, DefaultValueUpdateProcessorTest, PluginInfoTest, TestSort, TestPerFieldSimilarity, TestSerializedLuceneMatchVersion, AbstractAnalyticsFacetTest, CopyFieldTest, TestFoldingMultitermQuery, CoreAdminCreateDiscoverTest, TestCSVLoader, DocumentAnalysisRequestHandlerTest, SuggesterTSTTest, NumericFieldsTest, TestDocBasedVersionConstraints, TestBM25SimilarityFactory, SoftAutoCommitTest, TestMultiCoreConfBootstrap, TestCSVResponseWriter, TestFastLRUCache, SOLR749Test, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, SolrRequestParserTest, TestFieldResource, AbstractAnalyticsStatsTest, MultiTermTest, TestDefaultSearchFieldResource, HdfsSyncSliceTest, QueryResultKeyTest, CurrencyFieldOpenExchangeTest, UUIDFieldTest, TestDFRSimilarityFactory, TestQuerySenderListener, CurrencyFieldXmlFileTest, PathHierarchyTokenizerFactoryTest, LeaderElectionTest, TestNumberUtils, TestAddFieldRealTimeGet, FieldFacetTest, DeleteInactiveReplicaTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteInactiveReplicaTest -Dtests.seed=CAFADD75A6A985CD -Dtests.slow=true -Dtests.locale=en_AU -Dtests.timezone=Africa/Lome -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s | DeleteInactiveReplicaTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.DeleteInactiveReplicaTest: 
   [junit4]    >    1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CAFADD75A6A985CD]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=4306, name=Timer-275, state=WAITING, group=TGRP-DeleteInactiveReplicaTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:503)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:526)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:505)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CAFADD75A6A985CD]:0)
   [junit4] Completed in 49.26s, 1 test, 2 errors <<< FAILURES!

[...truncated 708 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1304: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:937: There were test failures: 358 suites, 1574 tests, 2 suite-level errors, 932 ignored

Total time: 53 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message