lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 8497 - Failure!
Date Fri, 06 Dec 2013 20:08:31 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8497/
Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([7F6D4107EEB202CD:7B65CEF4FC17EDEC]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:424)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:421)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:378)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:365)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:102)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 9961 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 1106015 T2627 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1386359650362
   [junit4]   2> 1106015 T2627 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1106019 T2627 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 1106020 T2627 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1106020 T2628 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1106120 T2627 oasc.ZkTestServer.run start zk server on port:54199
   [junit4]   2> 1106122 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1106124 T2634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@239bd707 name:ZooKeeperConnection Watcher:127.0.0.1:54199/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106125 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1106126 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1106127 T2636 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6106f3f3 name:ZooKeeperConnection Watcher:127.0.0.1:54199 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106127 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1106130 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1106132 T2638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c10ea88 name:ZooKeeperConnection Watcher:127.0.0.1:54199 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106132 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1106133 T2627 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1106140 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1106144 T2627 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1106148 T2627 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1106150 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1106154 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1106155 T2640 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42c0b955 name:ZooKeeperConnection Watcher:127.0.0.1:54199/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106156 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1106157 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1106162 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1106166 T2634 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1106167 T2640 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1106168 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1106169 T2642 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52da1f1c name:ZooKeeperConnection Watcher:127.0.0.1:54199/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106170 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1106171 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1106187 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1106193 T2627 oasc.Overseer.start Overseer (id=90856466046582788-127.0.0.1:54199_solr-n_0000000000) starting
   [junit4]   2> 1106196 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1106206 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1106216 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1106222 T2644 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1106225 T2643 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1106230 T2643 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1106232 T2643 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1106232 T2643 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 1106232 T2643 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1106239 T2642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1106240 T2634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1106241 T2640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1106729 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1106743 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1106751 T2642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1106753 T2627 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1106755 T2643 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1106758 T2640 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1106759 T2634 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1106760 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1106762 T2642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1106762 T2646 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d38de16 name:ZooKeeperConnection Watcher:127.0.0.1:54199/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1106762 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1106764 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1106769 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1106774 T2634 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1106774 T2646 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1106778 T2642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1106778 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1106781 T2643 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1106781 T2643 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1106785 T2642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1106791 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1106799 T2642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1106806 T2642 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1106910 T2634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1106910 T2646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1106914 T2627 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1106920 T2634 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1106921 T2646 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1106923 T2627 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54199 54199
   [junit4]   2> 1107144 T2627 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1107150 T2627 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 1107151 T2627 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1107151 T2647 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1107251 T2627 oasc.ZkTestServer.run start zk server on port:49623
   [junit4]   2> 1107253 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107255 T2653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f6f61e9 name:ZooKeeperConnection Watcher:127.0.0.1:49623/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107256 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107257 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107258 T2655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5535d75e name:ZooKeeperConnection Watcher:127.0.0.1:49623 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107259 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107300 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107302 T2657 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f5fb185 name:ZooKeeperConnection Watcher:127.0.0.1:49623 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107302 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107303 T2627 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1107311 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1107316 T2627 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1107320 T2627 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1107334 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1107344 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1107359 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107381 T2659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f18e82f name:ZooKeeperConnection Watcher:127.0.0.1:49623/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107381 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107383 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1107393 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1107397 T2627 oasc.Overseer.start Overseer (id=90856466120900611-127.0.0.1:49623_solr-n_0000000000) starting
   [junit4]   2> 1107401 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1107408 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1107412 T2661 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1107417 T2660 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1107418 T2660 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 1107419 T2660 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1107428 T2653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1107440 T2660 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1107442 T2653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1107447 T2660 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1107452 T2660 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1107453 T2660 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 1107457 T2659 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1107459 T2653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1107520 T2627 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49623 49623
   [junit4]   2> 1107692 T2627 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1107699 T2627 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
   [junit4]   2> 1107700 T2627 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1107701 T2662 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1107801 T2627 oasc.ZkTestServer.run start zk server on port:56044
   [junit4]   2> 1107805 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107807 T2668 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41a946e3 name:ZooKeeperConnection Watcher:127.0.0.1:56044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107808 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107809 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107810 T2670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2249434a name:ZooKeeperConnection Watcher:127.0.0.1:56044 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107811 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107846 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107847 T2672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7da52824 name:ZooKeeperConnection Watcher:127.0.0.1:56044 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107850 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107851 T2627 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1107856 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1107860 T2627 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1107864 T2627 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1107867 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1107872 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107874 T2674 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67f1dfde name:ZooKeeperConnection Watcher:127.0.0.1:56044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107874 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107886 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1107890 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1107895 T2674 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1107895 T2668 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1107896 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1107898 T2676 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7009eb98 name:ZooKeeperConnection Watcher:127.0.0.1:56044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1107898 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1107901 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1107916 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1107920 T2627 oasc.Overseer.start Overseer (id=90856466158059524-127.0.0.1:56044_solr-n_0000000000) starting
   [junit4]   2> 1107923 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1107932 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1107938 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1107942 T2678 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1107944 T2677 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1108945 T2676 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1108947 T2677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1108948 T2677 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1108948 T2677 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 1108948 T2677 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1108952 T2676 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1108953 T2674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1108953 T2668 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1109445 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1109457 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1109465 T2676 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109468 T2627 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1109470 T2677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1109475 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1109475 T2676 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109479 T2677 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1109480 T2677 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1109484 T2676 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1109596 T2668 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1109596 T2674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1110654 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1110665 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1110667 T2680 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@669c8a5c name:ZooKeeperConnection Watcher:127.0.0.1:56044/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1110667 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1110671 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1110681 T2627 oasc.Overseer.start Overseer (id=90856466158059525-127.0.0.1:56044_solr-n_0000000001) starting
   [junit4]   2> 1110688 T2682 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1110691 T2681 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1110697 T2681 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1110699 T2681 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 1110701 T2681 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1110709 T2680 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1110710 T2674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1110710 T2668 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1110765 T2680 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1110766 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1110770 T2681 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1110783 T2627 oasc.LeaderElector.checkIfIamLeader WARN Failed setting watch org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/90856466158059523-node1_core1-n_0000000002
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:269)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:126)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:597)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 
   [junit4]   2> 1110783 T2674 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1110787 T2681 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:169)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:166)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:166)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:650)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:645)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:645)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:645)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:883)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:226)
   [junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:187)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 	
   [junit4]   2> 1110787 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1110792 T2680 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1110801 T2668 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1110802 T2680 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1110803 T2674 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1110803 T2674 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1110807 T2627 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:56044 56044
   [junit4]   2> 1111175 T2627 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=7F6D4107EEB202CD -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=Europe/Vienna -Dtests.file.encoding=UTF-8
   [junit4] ERROR   3.50s J0 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7F6D4107EEB202CD:7B65CEF4FC17EDEC]:0)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:424)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:421)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:378)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:365)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:102)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 1111201 T2627 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 1111202 T2627 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1111202 T2683 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1111302 T2627 oasc.ZkTestServer.run start zk server on port:47497
   [junit4]   2> 1111303 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111307 T2689 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@293fd5d4 name:ZooKeeperConnection Watcher:127.0.0.1:47497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111307 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111341 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111342 T2691 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@744265c5 name:ZooKeeperConnection Watcher:127.0.0.1:47497 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111343 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111343 T2627 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1111348 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111349 T2693 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35084aea name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111349 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111350 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1111353 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111354 T2695 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4310e4e6 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111354 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111355 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1111362 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1111366 T2627 oasc.Overseer.start Overseer (id=90856466386714627-127.0.0.1:47497_solr-n_0000000000) starting
   [junit4]   2> 1111369 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1111385 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1111390 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1111394 T2697 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1111395 T2627 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1111397 T2696 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1111400 T2627 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1111403 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111407 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111409 T2699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ee04cef name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111409 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111411 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111415 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 1111421 T2693 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1111422 T2699 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 1111423 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111424 T2701 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@190b499a name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111424 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111427 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111432 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 1111437 T2699 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1111437 T2693 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1111438 T2701 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1111439 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111440 T2703 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131aa300 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111440 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111442 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111446 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 1111451 T2693 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111452 T2703 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111453 T2701 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111453 T2699 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1111453 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111454 T2705 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18e7171b name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111455 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111457 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111460 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 1111465 T2703 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1111466 T2693 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1111466 T2699 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1111467 T2701 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1111467 T2705 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1111468 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111469 T2707 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5810369 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111469 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111471 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111474 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 1111480 T2701 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1111480 T2693 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1111480 T2699 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1111480 T2703 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1111481 T2705 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1111481 T2707 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1111482 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111484 T2709 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@508316c7 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111484 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111486 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111489 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 1111495 T2703 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1111495 T2701 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1111496 T2693 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1111496 T2699 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1111496 T2709 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1111497 T2705 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1111498 T2707 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 1111498 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111499 T2711 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e7c8c9a name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111499 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111501 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111505 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 1111512 T2701 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111512 T2699 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111512 T2703 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111512 T2693 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111513 T2707 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111513 T2711 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111514 T2705 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111514 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111514 T2709 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 1111516 T2713 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c92a9e2 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111516 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111518 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111523 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 1111530 T2701 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111530 T2709 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111530 T2699 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111530 T2703 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111531 T2707 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111531 T2693 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111531 T2713 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111532 T2705 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111532 T2711 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 1111532 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111534 T2715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@204aa94c name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111534 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111535 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111542 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 1111557 T2711 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111558 T2709 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111558 T2701 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111558 T2693 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111559 T2699 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111559 T2703 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111560 T2715 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111561 T2707 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111563 T2713 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111563 T2705 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 1111563 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111565 T2717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c2d3caf name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111566 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111569 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111576 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 1111593 T2701 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111594 T2699 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111594 T2705 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111596 T2711 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111596 T2715 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111596 T2709 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111597 T2707 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111597 T2693 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111598 T2713 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111598 T2717 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111598 T2703 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 1111599 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111600 T2719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fc43ed1 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111601 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111602 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111605 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 1111617 T2713 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111617 T2705 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111617 T2717 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111618 T2709 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111618 T2703 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111618 T2699 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111618 T2715 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111619 T2719 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111619 T2707 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111619 T2693 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111620 T2701 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111620 T2711 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 1111621 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111633 T2721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71d5776a name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111634 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111635 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111639 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 1111647 T2709 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111648 T2699 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111648 T2705 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111649 T2713 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111649 T2711 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111649 T2715 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111650 T2707 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111650 T2719 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111650 T2703 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111650 T2717 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111651 T2721 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111652 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111652 T2701 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111652 T2693 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 1111653 T2723 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b423a6e name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111653 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111655 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111661 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 1111673 T2705 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111674 T2715 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111675 T2701 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111676 T2699 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111677 T2719 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111677 T2711 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111678 T2717 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111679 T2707 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111679 T2703 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111680 T2693 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111679 T2721 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111681 T2723 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111683 T2709 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111684 T2713 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 1111684 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111686 T2725 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@385a8b20 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111687 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111690 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111697 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 1111707 T2701 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111707 T2717 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111708 T2693 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111708 T2711 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111708 T2715 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111708 T2713 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111709 T2723 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111708 T2709 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111709 T2707 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111709 T2725 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111709 T2703 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111709 T2721 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111709 T2719 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111711 T2705 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111711 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111711 T2699 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 1111713 T2727 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6865f3cc name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111713 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111715 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111723 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 1111745 T2705 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111746 T2711 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111747 T2709 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111747 T2707 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111747 T2715 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111748 T2723 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111749 T2721 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111750 T2719 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111750 T2725 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111750 T2717 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111750 T2699 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111750 T2703 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111751 T2693 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111753 T2727 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111755 T2701 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111755 T2713 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 1111755 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111760 T2729 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22c202a9 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111761 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111763 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111770 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 1111779 T2707 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111781 T2701 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111782 T2709 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111782 T2715 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111782 T2721 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2705 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2727 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2717 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2719 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2725 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2699 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111784 T2713 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2703 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111783 T2693 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111784 T2729 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111786 T2711 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111786 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111786 T2723 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 1111788 T2731 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3490469f name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111788 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111790 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111794 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 1111806 T2701 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111807 T2709 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111808 T2715 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111808 T2707 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111808 T2723 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111808 T2727 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111808 T2721 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111809 T2719 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111809 T2703 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111809 T2717 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111810 T2725 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111810 T2729 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111810 T2699 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111810 T2711 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111810 T2693 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111811 T2731 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111812 T2713 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111813 T2705 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 1111813 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111814 T2733 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@595a89d4 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111814 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111816 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111820 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 1111833 T2715 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111834 T2727 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111835 T2701 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111836 T2693 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111837 T2711 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111838 T2699 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111839 T2731 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111839 T2709 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111839 T2707 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111840 T2723 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111840 T2721 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111840 T2719 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111841 T2717 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111841 T2703 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111841 T2729 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111841 T2725 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111843 T2733 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111843 T2713 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111844 T2705 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 1111845 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111847 T2735 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f783ace name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111847 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111851 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111857 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 1111899 T2701 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111899 T2711 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111900 T2715 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111900 T2729 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111900 T2719 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111901 T2705 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111901 T2723 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111902 T2721 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111902 T2733 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111902 T2717 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111902 T2699 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111902 T2703 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111903 T2693 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111903 T2731 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111903 T2713 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111902 T2725 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111903 T2735 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111904 T2707 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111905 T2709 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111905 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111905 T2727 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 1111907 T2737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ca71724 name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111907 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111908 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111912 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 1111930 T2701 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111930 T2693 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111931 T2725 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111931 T2731 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111931 T2717 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111932 T2713 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111932 T2715 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111932 T2709 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111932 T2723 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111933 T2707 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111933 T2721 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111933 T2727 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111933 T2719 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111934 T2703 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111934 T2733 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111934 T2699 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111934 T2735 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111934 T2729 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111936 T2737 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111938 T2711 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111938 T2705 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 1111939 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111942 T2739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11889dcd name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111942 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111945 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111953 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 1111966 T2701 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111967 T2713 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111967 T2709 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111969 T2725 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111969 T2733 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111969 T2729 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111970 T2719 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111970 T2737 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111970 T2693 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111970 T2731 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111970 T2711 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111971 T2707 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111971 T2721 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111971 T2727 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111972 T2735 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111971 T2723 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111972 T2717 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111972 T2699 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111972 T2703 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111973 T2739 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111974 T2705 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111974 T2715 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 1111974 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1111975 T2741 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29bfe70b name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1111976 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1111977 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1111980 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 1111993 T2701 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111993 T2731 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111994 T2709 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111994 T2715 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111996 T2705 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111996 T2733 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111996 T2739 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111996 T2717 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111996 T2735 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111997 T2699 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111997 T2703 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111997 T2711 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111997 T2725 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111998 T2693 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111998 T2741 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111999 T2737 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111999 T2723 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111999 T2729 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111999 T2727 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111999 T2721 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1111999 T2707 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1112001 T2713 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1112001 T2719 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 1112002 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1112003 T2743 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2be1c35e name:ZooKeeperConnection Watcher:127.0.0.1:47497/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1112004 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1112005 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1112008 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 1112018 T2723 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 1112025 T2713 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 1112025 T2703 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 1112025 T2725 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 1112025 T2721 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 1112026 T2741 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 1112026 T2711 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4

[...truncated too long message...]

ted type NodeChildrenChanged
   [junit4]   2> 1142922 T3152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1142960 T2627 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1142965 T3158 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1142969 T3159 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1142971 T3159 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 1142971 T3159 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 1142971 T3159 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 1142977 T3158 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1143083 T3152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1143123 T2627 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44731 44731
   [junit4]   2> 1143355 T2627 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1143356 T2627 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1143357 T2627 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1144037 T3145 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1144037 T3145 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1144038 T3145 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:252)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:249)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 1146357 T2627 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=es_HN, timezone=Europe/Vienna
   [junit4]   2> NOTE: Linux 3.8.0-34-generic amd64/Sun Microsystems Inc. 1.6.0_45 (64-bit)/cpus=8,threads=1,free=151280760,total=219074560
   [junit4]   2> NOTE: All tests run in this JVM: [TestSuggestSpellingConverter, BasicFunctionalityTest, BadCopyFieldTest, ClusterStateTest, UUIDFieldTest, LukeRequestHandlerTest, TestAnalyzeInfixSuggestions, RegexBoostProcessorTest, TestJmxMonitoredMap, TestRemoteStreaming, UpdateRequestProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestLRUCache, HdfsLockFactoryTest, TestUniqueKeyFieldResource, TestGroupingSearch, TestFaceting, TestPerFieldSimilarity, TestPHPSerializedResponseWriter, TestStressUserVersions, ResourceLoaderTest, CopyFieldTest, DistributedTermsComponentTest, TestSolrXml, DistributedQueryElevationComponentTest, TestSchemaNameResource, DeleteShardTest, LeaderElectionTest, SystemInfoHandlerTest, ReturnFieldsTest, TestDocSet, ScriptEngineTest, TestJoin, PrimitiveFieldTypeTest, TestCoreContainer, StatelessScriptUpdateProcessorFactoryTest, TestCopyFieldCollectionResource, TestPseudoReturnFields, TestPostingsSolrHighlighter, TestSolrDeletionPolicy1, URLClassifyProcessorTest, TestLMJelinekMercerSimilarityFactory, SimpleFacetsTest, QueryElevationComponentTest, DocumentBuilderTest, TestTrie, SyncSliceTest, ChangedSchemaMergeTest, CoreAdminCreateDiscoverTest, BlockCacheTest, TestSolrXMLSerializer, TestMergePolicyConfig, QueryResultKeyTest, UpdateParamsTest, JsonLoaderTest, TestFieldTypeCollectionResource, SynonymTokenizerTest, WordBreakSolrSpellCheckerTest, TestSurroundQueryParser, SolrCoreTest, TestSolrXmlPersistence, TestFileDictionaryLookup, TestValueSourceCache, TestMaxScoreQueryParser, ZkCLITest, DocValuesTest, LoggingHandlerTest, SoftAutoCommitTest, TestInfoStreamLogging, TestElisionMultitermQuery, InfoHandlerTest, DirectUpdateHandlerOptimizeTest, SpatialFilterTest, ExternalFileFieldSortTest, TestCloudManagedSchema, DateMathParserTest, DOMUtilTest, TestSchemaSimilarityResource, EchoParamsTest, SchemaVersionSpecificBehaviorTest, RecoveryZkTest, TestLuceneMatchVersion, TestLFUCache, IndexSchemaTest, TestLMDirichletSimilarityFactory, OverseerCollectionProcessorTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, OverseerTest]
   [junit4] Completed on J0 in 40.41s, 8 tests, 1 error <<< FAILURES!

[...truncated 523 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:426: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:406: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1307: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:940: There were test failures: 347 suites, 1516 tests, 1 error, 34 ignored (6 assumptions)

Total time: 59 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message