lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9901 - Still Failing!
Date Tue, 25 Mar 2014 11:15:42 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9901/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=1034, name=Overseer-91471531752816648-127.0.0.1:44816__fh-n_0000000010, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1034, name=Overseer-91471531752816648-127.0.0.1:44816__fh-n_0000000010, state=RUNNABLE, group=Overseer collection creation process.]
	at __randomizedtesting.SeedInfo.seed([F17FBEC930E6049:8EF175F4E4510075]:0)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([F17FBEC930E6049]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:274)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000006
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:276)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:273)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:206)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:257)
	... 1 more




Build Log:
[...truncated 10159 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 241546 T857 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_fh/
   [junit4]   2> 241549 T857 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1395744808238
   [junit4]   2> 241550 T857 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 241550 T858 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 241650 T857 oasc.ZkTestServer.run start zk server on port:43489
   [junit4]   2> 241651 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 241773 T864 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fe9875 name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 241774 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 241774 T857 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 241778 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 241779 T866 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c2636f name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 241779 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 241779 T857 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 241782 T857 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 241784 T857 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 241785 T857 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 241787 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 241787 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 241790 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 241791 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 241794 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 241794 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 241801 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 241801 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 241804 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 241804 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 241806 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 241806 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 241808 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 241808 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 241811 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 241811 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 241813 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 241813 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 241815 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 241816 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 241818 T857 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 241818 T857 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 242063 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 242066 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58292
   [junit4]   2> 242066 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 242066 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 242067 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509
   [junit4]   2> 242067 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509/'
   [junit4]   2> 242078 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509/solr.xml
   [junit4]   2> 242133 T857 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509
   [junit4]   2> 242134 T857 oasc.CoreContainer.<init> New CoreContainer 26698920
   [junit4]   2> 242134 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509/]
   [junit4]   2> 242135 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 242136 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 242136 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 242136 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 242137 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 242137 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 242137 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 242137 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 242138 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 242138 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 242139 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 242139 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 242139 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43489/solr
   [junit4]   2> 242140 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 242141 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242144 T877 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5164df name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242144 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242146 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242147 T879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c25810 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242147 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242149 T857 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 242151 T857 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 242153 T857 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 242156 T857 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 242157 T857 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 242159 T857 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 242161 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58292__fh
   [junit4]   2> 242163 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58292__fh
   [junit4]   2> 242165 T857 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 242168 T857 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 242171 T857 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58292__fh
   [junit4]   2> 242171 T857 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 242172 T857 oasc.Overseer.start Overseer (id=91471531752816643-127.0.0.1:58292__fh-n_0000000000) starting
   [junit4]   2> 242174 T857 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 242181 T881 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 242182 T857 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 242182 T881 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 242184 T857 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 242185 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 242193 T880 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 242194 T857 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509
   [junit4]   2> 242200 T857 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509/conf/
   [junit4]   2> 242321 T857 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 242322 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 242322 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 242323 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242325 T884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@117d777 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242325 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242327 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 242328 T857 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 242587 T857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 242587 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 242589 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54225
   [junit4]   2> 242590 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 242590 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 242591 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017
   [junit4]   2> 242591 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/'
   [junit4]   2> 242603 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/solr.xml
   [junit4]   2> 242652 T857 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017
   [junit4]   2> 242652 T857 oasc.CoreContainer.<init> New CoreContainer 27725671
   [junit4]   2> 242653 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/]
   [junit4]   2> 242654 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 242654 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 242654 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 242655 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 242655 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 242655 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 242656 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 242656 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 242656 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 242657 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 242657 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 242657 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 242658 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43489/solr
   [junit4]   2> 242658 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 242659 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242660 T895 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e45b05 name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242660 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242663 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 242664 T897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4dc395 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 242664 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 242668 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 243671 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54225__fh
   [junit4]   2> 243672 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54225__fh
   [junit4]   2> 243674 T879 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 243674 T884 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 243674 T897 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 243680 T857 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017
   [junit4]   2> 243686 T857 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/conf/
   [junit4]   2> 243809 T857 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 243810 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 243810 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 244042 T857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 244042 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 244044 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44816
   [junit4]   2> 244045 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 244045 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 244046 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499
   [junit4]   2> 244046 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499/'
   [junit4]   2> 244059 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499/solr.xml
   [junit4]   2> 244118 T857 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499
   [junit4]   2> 244119 T857 oasc.CoreContainer.<init> New CoreContainer 13054120
   [junit4]   2> 244119 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499/]
   [junit4]   2> 244121 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 244121 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 244122 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 244122 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 244123 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 244123 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 244123 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 244124 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 244124 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 244125 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 244125 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 244126 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 244126 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43489/solr
   [junit4]   2> 244127 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 244128 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 244129 T909 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14b80aa name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 244130 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 244131 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 244133 T911 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5476b6 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 244133 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 244138 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 245140 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44816__fh
   [junit4]   2> 245142 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44816__fh
   [junit4]   2> 245147 T884 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245147 T879 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245147 T911 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245147 T897 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 245155 T857 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499
   [junit4]   2> 245165 T857 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499/conf/
   [junit4]   2> 245283 T857 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 245284 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 245284 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 245529 T857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 245529 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 245531 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50199
   [junit4]   2> 245532 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 245532 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 245532 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973
   [junit4]   2> 245533 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973/'
   [junit4]   2> 245544 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973/solr.xml
   [junit4]   2> 245594 T857 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973
   [junit4]   2> 245594 T857 oasc.CoreContainer.<init> New CoreContainer 20556877
   [junit4]   2> 245595 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973/]
   [junit4]   2> 245596 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 245596 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 245596 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 245597 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 245597 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 245597 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 245598 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 245598 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 245598 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 245599 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 245599 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 245599 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 245600 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43489/solr
   [junit4]   2> 245600 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 245600 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 245602 T923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15f1ce4 name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 245602 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 245604 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 245605 T925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d490 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 245606 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 245611 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 246613 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50199__fh
   [junit4]   2> 246614 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50199__fh
   [junit4]   2> 246616 T911 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 246617 T879 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 246617 T884 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 246617 T897 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 246617 T925 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 246622 T857 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973
   [junit4]   2> 246628 T857 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973/conf/
   [junit4]   2> 246750 T857 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 246751 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 246751 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 246994 T857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 246995 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 246997 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43737
   [junit4]   2> 246997 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 246998 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 246998 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440
   [junit4]   2> 246999 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440/'
   [junit4]   2> 247017 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440/solr.xml
   [junit4]   2> 247083 T857 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440
   [junit4]   2> 247083 T857 oasc.CoreContainer.<init> New CoreContainer 2120319
   [junit4]   2> 247084 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440/]
   [junit4]   2> 247085 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 247085 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 247085 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 247086 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 247086 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 247086 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 247086 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 247087 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 247087 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 247088 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 247088 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 247088 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 247088 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43489/solr
   [junit4]   2> 247089 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 247089 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 247091 T937 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1548f0 name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 247091 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 247093 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 247094 T939 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a15ae5 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 247094 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 247099 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 248101 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43737__fh
   [junit4]   2> 248102 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43737__fh
   [junit4]   2> 248106 T911 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 248106 T884 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 248106 T925 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 248106 T897 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 248106 T879 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 248106 T939 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 248110 T857 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440
   [junit4]   2> 248115 T857 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440/conf/
   [junit4]   2> 248211 T857 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 248212 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 248213 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 248451 T857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 248452 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 248454 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60168
   [junit4]   2> 248454 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 248455 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 248455 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901
   [junit4]   2> 248455 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901/'
   [junit4]   2> 248472 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901/solr.xml
   [junit4]   2> 248520 T857 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901
   [junit4]   2> 248521 T857 oasc.CoreContainer.<init> New CoreContainer 28259709
   [junit4]   2> 248521 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901/]
   [junit4]   2> 248522 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 248523 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 248523 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 248523 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 248523 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 248524 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 248524 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 248524 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 248525 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 248525 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 248525 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 248526 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 248526 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43489/solr
   [junit4]   2> 248526 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 248527 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 248528 T951 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e0b0c4 name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 248529 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 248532 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 248534 T953 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131dfd9 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 248534 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 248539 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 249541 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60168__fh
   [junit4]   2> 249542 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60168__fh
   [junit4]   2> 249545 T879 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 249545 T953 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 249545 T911 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 249545 T939 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 249545 T884 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 249545 T897 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 249546 T925 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 249550 T857 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901
   [junit4]   2> 249555 T857 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901/conf/
   [junit4]   2> 249650 T857 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 249651 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 249651 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 249903 T857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 249904 T857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 249906 T857 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42431
   [junit4]   2> 249906 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 249907 T857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 249907 T857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340
   [junit4]   2> 249907 T857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/'
   [junit4]   2> 249918 T857 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/solr.xml
   [junit4]   2> 249965 T857 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340
   [junit4]   2> 249965 T857 oasc.CoreContainer.<init> New CoreContainer 26197740
   [junit4]   2> 249965 T857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/]
   [junit4]   2> 249967 T857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 249967 T857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 249967 T857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 249967 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 249968 T857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 249968 T857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 249968 T857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 249968 T857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 249969 T857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 249969 T857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 249969 T857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 249970 T857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 249970 T857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43489/solr
   [junit4]   2> 249970 T857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 249971 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 249972 T965 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f78c1 name:ZooKeeperConnection Watcher:127.0.0.1:43489 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 249973 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 249975 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 249976 T967 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9956cb name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 249976 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 249981 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 250983 T857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42431__fh
   [junit4]   2> 250984 T857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42431__fh
   [junit4]   2> 250986 T911 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250987 T879 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250987 T939 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250987 T967 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250987 T897 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250987 T953 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250987 T925 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250987 T884 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 250992 T857 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340
   [junit4]   2> 251001 T857 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/conf/
   [junit4]   2> 251157 T857 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 251159 T857 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 251159 T857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 251161 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 251162 T970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11d40bc name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 251162 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 251163 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 251166 T857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 251167 T972 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b36d74 name:ZooKeeperConnection Watcher:127.0.0.1:43489/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 251167 T857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 251168 T857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 251173 T893 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&name=testOverseerCol&action=CREATE&version=2&wt=javabin&numShards=4&maxShardsPerNode=2
   [junit4]   2> 251175 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251176 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251176 T881 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 251176 T881 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 251177 T881 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 251178 T881 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 251178 T881 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 251180 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251181 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 251181 T880 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 251182 T880 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 251183 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251184 T970 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T939 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T953 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T897 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251184 T967 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251280 T881 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection, shardNames [shard1, shard2, shard3, shard4] , replicationFactor : 2
   [junit4]   2> 251280 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:42431__fh
   [junit4]   2> 251281 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:50199__fh
   [junit4]   2> 251282 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:54225__fh
   [junit4]   2> 251283 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:58292__fh
   [junit4]   2> 251283 T963 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard1_replica1&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard1&numShards=4
   [junit4]   2> 251284 T921 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard1_replica2&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard1&numShards=4
   [junit4]   2> 251285 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:43737__fh
   [junit4]   2> 251285 T890 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard2_replica1&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard2&numShards=4
   [junit4]   2> 251286 T873 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard2_replica2&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard2&numShards=4
   [junit4]   2> 251286 T963 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down collection=testOverseerCol
   [junit4]   2> 251287 T890 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down collection=testOverseerCol
   [junit4]   2> 251286 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:44816__fh
   [junit4]   2> 251288 T933 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard3_replica1&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard3&numShards=4
   [junit4]   2> 251288 T921 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down collection=testOverseerCol
   [junit4]   2> 251289 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251289 T963 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251290 T921 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251289 T873 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down collection=testOverseerCol
   [junit4]   2> 251289 T890 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251290 T880 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 251290 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:60168__fh
   [junit4]   2> 251290 T933 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down collection=testOverseerCol
   [junit4]   2> 251292 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42431/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42431__fh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251291 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251291 T873 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251291 T904 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard3_replica2&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard3&numShards=4
   [junit4]   2> 251294 T949 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard4_replica1&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard4&numShards=4
   [junit4]   2> 251293 T933 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251292 T881 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:42431__fh
   [junit4]   2> 251295 T904 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down collection=testOverseerCol
   [junit4]   2> 251295 T949 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down collection=testOverseerCol
   [junit4]   2> 251296 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54225/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54225__fh",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251296 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251296 T904 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251297 T949 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251297 T962 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&name=testOverseerCol_shard4_replica2&collection.configName=conf1&action=CREATE&version=2&wt=javabin&collection=testOverseerCol&shard=shard4&numShards=4
   [junit4]   2> 251299 T962 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down collection=testOverseerCol
   [junit4]   2> 251300 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251300 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50199/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50199__fh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251300 T962 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 251302 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58292/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58292__fh",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251311 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251312 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43737/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:43737__fh",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251313 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251314 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44816/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44816__fh",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251316 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251316 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:60168/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:60168__fh",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251318 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251318 T880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42431/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42431__fh",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 251320 T879 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 251423 T970 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251423 T967 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251423 T972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251423 T879 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251423 T939 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251423 T897 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251424 T925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251424 T884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251423 T953 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 251423 T911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 252290 T963 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 252290 T921 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 252291 T890 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 252291 T921 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252291 T963 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252292 T890 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252292 T921 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252292 T963 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252292 T890 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252293 T890 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252292 T921 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252294 T873 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 252294 T890 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252295 T921 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252293 T963 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252295 T921 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1395744811973/testOverseerCol_shard1_replica2/'
   [junit4]   2> 252295 T873 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252295 T890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/testOverseerCol_shard2_replica1/'
   [junit4]   2> 252297 T873 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252297 T873 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252295 T933 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 252297 T904 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 252296 T963 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252298 T933 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252298 T873 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252299 T904 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252298 T963 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/testOverseerCol_shard1_replica1/'
   [junit4]   2> 252300 T949 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 252300 T873 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1395744808509/testOverseerCol_shard2_replica2/'
   [junit4]   2> 252301 T904 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252302 T904 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252301 T962 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 252302 T933 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252303 T933 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252301 T949 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252304 T904 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252305 T904 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1395744810499/testOverseerCol_shard3_replica2/'
   [junit4]   2> 252303 T962 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 252308 T933 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252308 T962 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252308 T933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty4-1395744813440/testOverseerCol_shard3_replica1/'
   [junit4]   2> 252309 T949 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 252309 T949 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252309 T962 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 252311 T949 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252312 T949 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1395744814901/testOverseerCol_shard4_replica1/'
   [junit4]   2> 252312 T962 oascc.ZkStateReader.readConfigName path=/collections/testOverseerCol configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 252313 T962 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/testOverseerCol_shard4_replica2/'
   [junit4]   2> 252387 T921 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252413 T890 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252415 T933 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252471 T904 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252472 T873 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252491 T963 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252509 T949 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252512 T962 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 252528 T921 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252534 T921 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252555 T921 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 252590 T933 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252596 T933 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252597 T890 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252598 T890 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252614 T873 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252614 T933 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 252619 T890 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 252626 T963 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252627 T963 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252629 T873 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252642 T873 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 252653 T963 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 252656 T962 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252657 T962 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252675 T962 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 252676 T904 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252677 T949 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 252679 T904 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252679 T949 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 252698 T904 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 252701 T949 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 253882 T890 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 253922 T890 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 253927 T890 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 253937 T962 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 253953 T962 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 253959 T962 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 253961 T949 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 253965 T963 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 253966 T890 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 253980 T890 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 253981 T963 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 253982 T962 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 253982 T949 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 253982 T921 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 253986 T962 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 253987 T963 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 253988 T949 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 253986 T890 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 253993 T890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 253995 T890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 253995 T921 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 253996 T890 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 254003 T890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 254003 T890 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 254003 T962 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 254004 T890 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using configuration from collection testOverseerCol
   [junit4]   2> 254004 T890 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 254005 T890 oasc.SolrCore.<init> [testOverseerCol_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/testOverseerCol_shard2_replica1/, dataDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/testOverseerCol_shard2_replica1/data/
   [junit4]   2> 254005 T890 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16fd193
   [junit4]   2> 254006 T890 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/testOverseerCol_shard2_replica1/data
   [junit4]   2> 254007 T890 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/testOverseerCol_shard2_replica1/data/index/
   [junit4]   2> 254007 T921 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 254007 T890 oasc.SolrCore.initIndex WARN [testOverseerCol_shard2_replica1] Solr index directory '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/testOverseerCol_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 254008 T890 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1395744809017/testOverseerCol_shard2_replica1/data/index
   [junit4]   2> 254009 T890 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=20, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 254010 T963 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 254010 T962 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 254011 T962 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 254011 T962 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 254010 T890 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a24b31 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1447bdc),segFN=segments_1,generation=1}
   [junit4]   2> 254012 T890 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 254013 T962 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 254013 T962 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 254014 T962 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using configuration from collection testOverseerCol
   [junit4]   2> 254014 T962 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 254014 T962 oasc.SolrCore.<init> [testOverseerCol_shard4_replica2] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/testOverseerCol_shard4_replica2/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/testOverseerCol_shard4_replica2/data/
   [junit4]   2> 254015 T962 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16fd193
   [junit4]   2> 254015 T963 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 254021 T962 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/testOverseerCol_shard4_replica2/data
   [junit4]   2> 254021 T963 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 254022 T962 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/testOverseerCol_shard4_replica2/data/index/
   [junit4]   2> 254023 T962 oasc.SolrCore.initIndex WARN [testOverseerCol_shard4_replica2] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-jetty6-1395744816340/testOverseerCol_shard4_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 254024 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 254024 T963 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 254022 T890 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 254025 T963 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4] 

[...truncated too long message...]

base_url":"http://127.0.0.1:42431/_fh",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:42431__fh",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node8"}
   [junit4]   2> 265969 T1032 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 265982 T857 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 265983 T857 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 265983 T857 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 265983 T857 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 265984 T857 oasc.SolrCore.closeSearcher [testOverseerCol_shard1_replica1] Closing main searcher on request.
   [junit4]   2> 265985 T857 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 265985 T857 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6;done=false>>]
   [junit4]   2> 265985 T857 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6
   [junit4]   2> 265986 T857 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6/index;done=false>>]
   [junit4]   2> 265986 T857 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6/index
   [junit4]   2> 265986 T857 oasc.SolrCore.close [testOverseerCol_shard4_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@1f137a7
   [junit4]   2> 266010 T857 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 266010 T857 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 266011 T857 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 266011 T857 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 266012 T857 oasc.SolrCore.closeSearcher [testOverseerCol_shard4_replica2] Closing main searcher on request.
   [junit4]   2> 266012 T857 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 266012 T857 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6;done=false>>]
   [junit4]   2> 266013 T857 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6
   [junit4]   2> 266013 T857 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6/index;done=false>>]
   [junit4]   2> 266013 T857 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.OverseerRolesTest-1395744808238/jetty6/index
   [junit4]   2> 266016 T884 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 266016 T970 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 266037 T857 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_fh,null}
   [junit4]   2> 266091 T857 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58292
   [junit4]   2> 266093 T857 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 266095 T857 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43489 43489
   [junit4]   2> 266396 T939 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 267515 T1032 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 266488 T953 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerRolesTest -Dtests.method=testDistribSearch -Dtests.seed=F17FBEC930E6049 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_IQ -Dtests.timezone=Asia/Riyadh87 -Dtests.file.encoding=UTF-8
   [junit4] ERROR   34.6s J0 | OverseerRolesTest.testDistribSearch <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1034, name=Overseer-91471531752816648-127.0.0.1:44816__fh-n_0000000010, state=RUNNABLE, group=Overseer collection creation process.]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F17FBEC930E6049:8EF175F4E4510075]:0)
   [junit4]    > Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F17FBEC930E6049]:0)
   [junit4]    > 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:274)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000006
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:276)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:273)
   [junit4]    > 	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:206)
   [junit4]    > 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:257)
   [junit4]    > 	... 1 more
   [junit4]   2> 276175 T857 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_IQ, timezone=Asia/Riyadh87
   [junit4]   2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=27105752,total=96645120
   [junit4]   2> NOTE: All tests run in this JVM: [TestAddFieldRealTimeGet, TestManagedResource, LukeRequestHandlerTest, TestWriterPerf, UnloadDistributedZkTest, SOLR749Test, SynonymTokenizerTest, SimpleFacetsTest, DistributedExpandComponentTest, DistributedQueryElevationComponentTest, TestStressVersions, TestFreeTextSuggestions, PeerSyncTest, SliceStateUpdateTest, TestUniqueKeyFieldResource, TestRemoteStreaming, AutoCommitTest, OverseerCollectionProcessorTest, DeleteShardTest, TestSolrXmlPersistor, CopyFieldTest, TestRealTimeGet, MBeansHandlerTest, PolyFieldTest, TestInitQParser, FileBasedSpellCheckerTest, TestLRUCache, TestExtendedDismaxParser, SearchHandlerTest, DistributedQueryComponentCustomSortTest, CoreMergeIndexesAdminHandlerTest, TestSolrXMLSerializer, TestImplicitCoreProperties, TestManagedResourceStorage, TestPerFieldSimilarity, OverseerRolesTest]
   [junit4] Completed on J0 in 34.66s, 1 test, 1 error <<< FAILURES!

[...truncated 1027 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1275: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 393 suites, 1650 tests, 1 error, 49 ignored (28 assumptions)

Total time: 60 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message