lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60-ea-b15) - Build # 10212 - Failure!
Date Sun, 18 May 2014 07:08:29 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10212/
Java: 32bit/jdk1.7.0_60-ea-b15 -client -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([B7625CA50F8D8263:B36AD3561D286D42]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
	... 47 more
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
	... 48 more




Build Log:
[...truncated 11466 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/solr/build/solr-core/test/J1/./solr.cloud.OverseerTest-B7625CA50F8D8263-001/init-core-data-001
   [junit4]   2> 975373 T3552 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 975374 T3552 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 975374 T3552 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 975377 T3552 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 975378 T3552 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 975379 T3553 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 975479 T3552 oasc.ZkTestServer.run start zk server on port:35614
   [junit4]   2> 975481 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 975484 T3559 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@db113f name:ZooKeeperConnection Watcher:127.0.0.1:35614 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 975485 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 975489 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 975491 T3561 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@db1a43 name:ZooKeeperConnection Watcher:127.0.0.1:35614 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 975492 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 975492 T3552 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 975498 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 975500 T3563 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13166f1 name:ZooKeeperConnection Watcher:127.0.0.1:35614/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 975501 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 975501 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 975505 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 975507 T3565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17a0bae name:ZooKeeperConnection Watcher:127.0.0.1:35614/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 975508 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 975510 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 975513 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 975519 T3552 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35614_solr
   [junit4]   2> 975520 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 975523 T3552 oasc.Overseer.start Overseer (id=91776367132672003-127.0.0.1:35614_solr-n_0000000000) starting
   [junit4]   2> 975526 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 975530 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 975535 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 975540 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 975544 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 975549 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 975556 T3567 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 975557 T3566 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 975557 T3552 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 975562 T3552 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 975564 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 975572 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 975574 T3569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18b9705 name:ZooKeeperConnection Watcher:127.0.0.1:35614/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 975574 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 975577 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 975580 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 975584 T3569 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 975584 T3563 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 975586 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 975588 T3566 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 975589 T3566 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 975590 T3566 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 975590 T3566 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 975594 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 975596 T3569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 975596 T3563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 976087 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 976099 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 976106 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976109 T3566 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 976110 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976114 T3566 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 976115 T3566 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 976115 T3566 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 976118 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976221 T3569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 976221 T3563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 976611 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 976620 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 976626 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976627 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976627 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976629 T3566 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 976631 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976635 T3566 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 976636 T3566 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 976636 T3566 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 976640 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 976743 T3569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 976743 T3563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 977131 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 977140 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 977148 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977148 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977149 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977151 T3566 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 977155 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977156 T3566 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 977157 T3566 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 977157 T3566 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 977161 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977264 T3563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 977264 T3569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 977650 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 977655 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977656 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977656 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977657 T3566 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 977657 T3566 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 977657 T3566 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 977658 T3566 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 977659 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 977763 T3569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 977763 T3563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 978156 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 978165 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 978166 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 978166 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 978167 T3566 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 978168 T3566 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 978169 T3566 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 978169 T3566 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 978173 T3565 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 978276 T3569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 978276 T3563 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 978665 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 978677 T3569 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 978680 T3552 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35614 35614
   [junit4]   2> 979589 T3553 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35614 35614
   [junit4]   2> 979592 T3552 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 979593 T3566 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:35614_solr
   [junit4]   2> 979595 T3570 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 979601 T3552 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 979602 T3552 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 979604 T3571 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 979704 T3552 oasc.ZkTestServer.run start zk server on port:44352
   [junit4]   2> 979705 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 979710 T3577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cc8286 name:ZooKeeperConnection Watcher:127.0.0.1:44352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 979711 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 979712 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 979714 T3579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d513ec name:ZooKeeperConnection Watcher:127.0.0.1:44352 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 979714 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 979719 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 979720 T3581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11e1345 name:ZooKeeperConnection Watcher:127.0.0.1:44352 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 979721 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 979721 T3552 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 979730 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 979734 T3552 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 979737 T3552 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 979740 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 979744 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 979746 T3583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b7c13 name:ZooKeeperConnection Watcher:127.0.0.1:44352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 979746 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 979749 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 979753 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 979756 T3583 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 979756 T3577 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 979757 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 979758 T3585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a51a73 name:ZooKeeperConnection Watcher:127.0.0.1:44352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 979759 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 979760 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 979764 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 979771 T3552 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44352_solr
   [junit4]   2> 979771 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 979775 T3552 oasc.Overseer.start Overseer (id=91776367409561604-127.0.0.1:44352_solr-n_0000000000) starting
   [junit4]   2> 979778 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 979783 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 979788 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 979792 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 979797 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 979804 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 979813 T3587 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 979814 T3586 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 979817 T3586 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 979818 T3586 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 979818 T3586 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 979819 T3586 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 979822 T3585 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 979823 T3583 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 979823 T3577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 980177 T3569 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 980179 T3569 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 980180 T3569 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 980315 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 980328 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 980335 T3585 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 980336 T3552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 980337 T3586 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 980339 T3577 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 980339 T3583 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 980342 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 980343 T3585 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 980344 T3589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aac096 name:ZooKeeperConnection Watcher:127.0.0.1:44352/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 980345 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 980347 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 980350 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 980353 T3577 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 980353 T3589 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 980356 T3585 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 980356 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 980358 T3586 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 980358 T3586 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 980361 T3585 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 980364 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 980369 T3585 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 980374 T3585 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 980477 T3577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 980477 T3589 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 980478 T3552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 980483 T3577 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 980483 T3589 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 980486 T3552 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44352 44352
   [junit4]   2> 981171 T3571 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44352 44352
   [junit4]   2> 981175 T3552 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 981176 T3586 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44352_solr
   [junit4]   2> 981177 T3590 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 981184 T3552 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 981185 T3552 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 981186 T3591 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 981286 T3552 oasc.ZkTestServer.run start zk server on port:46813
   [junit4]   2> 981287 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 981291 T3597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b03b4d name:ZooKeeperConnection Watcher:127.0.0.1:46813/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 981292 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 981293 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 981295 T3599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fb96ad name:ZooKeeperConnection Watcher:127.0.0.1:46813 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 981295 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 981299 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 981301 T3601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@940699 name:ZooKeeperConnection Watcher:127.0.0.1:46813 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 981302 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 981302 T3552 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 981307 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 981310 T3552 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 981313 T3552 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 981316 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 981322 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 981328 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 981330 T3603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49c810 name:ZooKeeperConnection Watcher:127.0.0.1:46813/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 981330 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 981332 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 981335 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 981342 T3552 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46813_solr
   [junit4]   2> 981342 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 981345 T3552 oasc.Overseer.start Overseer (id=91776367513370627-127.0.0.1:46813_solr-n_0000000000) starting
   [junit4]   2> 981348 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 981355 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 981359 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 981363 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 981367 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 981374 T3605 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 981376 T3604 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 981377 T3604 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 981378 T3604 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 981380 T3597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 981383 T3604 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 981384 T3597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 981386 T3604 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 981388 T3604 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 981389 T3604 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 981391 T3603 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 981392 T3597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 981480 T3552 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46813 46813
   [junit4]   2> 982327 T3591 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46813 46813
   [junit4]   2> 982329 T3552 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   2> 982330 T3604 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:46813_solr
   [junit4]   2> 982331 T3606 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 982336 T3552 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
   [junit4]   2> 982337 T3552 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 982338 T3607 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 982437 T3552 oasc.ZkTestServer.run start zk server on port:50760
   [junit4]   2> 982439 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 982441 T3613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c735d name:ZooKeeperConnection Watcher:127.0.0.1:50760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 982442 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 982442 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 982443 T3615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d4649 name:ZooKeeperConnection Watcher:127.0.0.1:50760 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 982444 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 982446 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 982446 T3617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7632ad name:ZooKeeperConnection Watcher:127.0.0.1:50760 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 982447 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 982447 T3552 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 982449 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 982451 T3552 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 982452 T3552 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 982453 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 982455 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 982456 T3619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1728fe4 name:ZooKeeperConnection Watcher:127.0.0.1:50760/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 982456 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 982456 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 982458 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 982460 T3552 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50760_solr
   [junit4]   2> 982461 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 982462 T3552 oasc.Overseer.start Overseer (id=91776367588737027-127.0.0.1:50760_solr-n_0000000000) starting
   [junit4]   2> 982463 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 982465 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 982466 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 982470 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 982474 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 982477 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 982481 T3621 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 982481 T3620 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 982483 T3552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 982486 T3620 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 982487 T3620 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 982488 T3620 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 982492 T3619 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 982492 T3613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 982534 T3552 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 982542 T3619 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 982544 T3620 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 982545 T3620 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 982546 T3620 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 982547 T3620 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 982562 T3619 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 982665 T3613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 982695 T3552 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50760 50760
   [junit4]   2> 983377 T3607 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50760 50760
   [junit4]   2> 983379 T3552 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   2> 983380 T3620 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:50760_solr
   [junit4]   2> 983380 T3622 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 983384 T3552 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 983385 T3552 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 983385 T3623 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 983485 T3552 oasc.ZkTestServer.run start zk server on port:53721
   [junit4]   2> 983486 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983499 T3629 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ec937 name:ZooKeeperConnection Watcher:127.0.0.1:53721 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983499 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983504 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983505 T3631 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13ec438 name:ZooKeeperConnection Watcher:127.0.0.1:53721 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983506 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983506 T3552 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 983511 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983513 T3633 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c7c26 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983513 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983513 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 983516 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983518 T3635 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b26af1 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983518 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983520 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 983522 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 983524 T3552 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53721_solr
   [junit4]   2> 983525 T3552 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 983526 T3552 oasc.Overseer.start Overseer (id=91776367657353219-127.0.0.1:53721_solr-n_0000000000) starting
   [junit4]   2> 983527 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 983529 T3552 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 983530 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 983532 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 983534 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 983536 T3552 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 983541 T3637 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 983541 T3552 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 983541 T3636 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 983542 T3552 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 983543 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983545 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983546 T3639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c58106 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983546 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983547 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983549 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 983551 T3639 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 983551 T3633 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 983551 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983552 T3641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f908a name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983552 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983553 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983554 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 983556 T3639 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 983556 T3633 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 983557 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983556 T3641 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 983557 T3643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a8a54 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983557 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983559 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983560 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 983562 T3633 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 983562 T3643 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 983562 T3639 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 983562 T3641 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 983562 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983563 T3645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b4f9ff name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983563 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983564 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983566 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 983567 T3633 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 983568 T3639 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 983567 T3643 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 983568 T3641 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 983568 T3645 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 983568 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983570 T3647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@349a19 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983571 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983573 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983575 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 983579 T3633 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 983579 T3641 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 983579 T3639 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 983580 T3643 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 983580 T3645 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 983586 T3647 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 983587 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983588 T3649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3604f4 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983588 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983590 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983592 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 983595 T3645 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 983595 T3633 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 983596 T3641 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 983596 T3647 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 983595 T3649 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 983595 T3639 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 983595 T3643 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 983596 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983597 T3651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f1d69d name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983598 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983599 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983602 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 983604 T3641 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983604 T3645 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983605 T3643 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983605 T3651 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983605 T3647 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983605 T3639 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983605 T3649 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983605 T3633 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 983605 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983606 T3653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c65bd0 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983606 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983607 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983609 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 983610 T3641 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983610 T3647 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983611 T3639 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983611 T3653 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983611 T3651 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983611 T3643 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983610 T3645 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983611 T3649 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983610 T3633 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 983611 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983612 T3655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d187b8 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983613 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983614 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983615 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 983617 T3645 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3647 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3633 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3655 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3649 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3653 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983618 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983617 T3639 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3643 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3651 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983617 T3641 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 983619 T3657 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@145314e name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983620 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983621 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983622 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 983624 T3649 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983625 T3651 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983625 T3647 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983624 T3655 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983624 T3645 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983625 T3657 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983625 T3643 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983624 T3641 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983624 T3633 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983624 T3639 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983624 T3653 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 983626 T3659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f164 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983625 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983627 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983628 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983630 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 983631 T3641 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983631 T3633 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983632 T3643 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983632 T3647 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983632 T3649 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983631 T3645 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983631 T3639 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983633 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983632 T3657 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983632 T3659 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983632 T3655 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983632 T3653 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983632 T3651 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 983633 T3661 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1898e25 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983634 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983636 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983637 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 983640 T3633 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983640 T3643 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3639 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983640 T3645 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983640 T3641 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983640 T3649 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983641 T3647 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3653 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3655 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3661 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3659 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3651 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983641 T3657 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 983642 T3663 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12ea060 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983643 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983644 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983647 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 983650 T3641 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3649 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3657 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3653 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3659 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3633 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3643 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3651 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3655 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3639 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3645 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983653 T3665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cc47cc name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983652 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983651 T3663 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3661 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983651 T3647 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 983655 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983657 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983659 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 983661 T3641 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983661 T3639 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3665 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3655 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3645 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983663 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983662 T3649 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3647 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3653 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3633 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3643 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3657 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983661 T3651 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983661 T3663 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983665 T3667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@986f36 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983662 T3661 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983662 T3659 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 983667 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983669 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983672 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 983674 T3651 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983674 T3665 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983674 T3641 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3661 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3667 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3643 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3659 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3647 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3645 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3655 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3633 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3639 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3653 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983675 T3663 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983674 T3649 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983674 T3657 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 983676 T3669 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e2f6c name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983676 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983678 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983679 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983681 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 983683 T3651 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983683 T3657 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3665 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3653 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3655 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983683 T3643 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983683 T3649 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983684 T3669 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3633 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3661 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3659 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3647 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3667 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3645 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3639 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983684 T3663 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983683 T3641 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 983685 T3671 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e77597 name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983687 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983688 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983690 T3552 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 983692 T3641 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983692 T3665 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3643 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3661 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3645 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983694 T3647 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983694 T3653 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3663 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3651 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3639 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3649 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3633 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983695 T3673 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a2da6d name:ZooKeeperConnection Watcher:127.0.0.1:53721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 983694 T3552 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 983694 T3671 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983694 T3659 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983694 T3667 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3655 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3669 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983693 T3657 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 983696 T3552 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 983698 T3552 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 983699 T3552 oascc.SolrZkClient.makePath makePath: /live_node

[...truncated too long message...]

akePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1025067 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1026071 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1027076 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1028081 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1029086 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1030089 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1031095 T3552 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1031101 T4062 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1031101 T4064 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 1031105 T3552 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39528 39528
   [junit4]   2> 1031809 T4052 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39528 39528
   [junit4]   2> 1031812 T3552 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
   [junit4]   2> 1031813 T4072 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:39528_solr
   [junit4]   2> 1031814 T4074 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=B7625CA50F8D8263 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_ZA -Dtests.timezone=Australia/Darwin -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   18.8s J1 | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B7625CA50F8D8263:B36AD3561D286D42]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
   [junit4]    > 	... 47 more
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
   [junit4]    > 	... 48 more
   [junit4]   2> 1031832 T3552 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 1031833 T3552 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 1034833 T3552 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/solr/build/solr-core/test/J1/./solr.cloud.OverseerTest-B7625CA50F8D8263-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=en_ZA, timezone=Australia/Darwin
   [junit4]   2> NOTE: Linux 3.8.0-39-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=99032256,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [SpellCheckComponentTest, TestStressReorder, CurrencyFieldOpenExchangeTest, TestFastWriter, CoreMergeIndexesAdminHandlerTest, XmlUpdateRequestHandlerTest, FullSolrCloudDistribCmdsTest, AliasIntegrationTest, TestSimpleQParserPlugin, PingRequestHandlerTest, SpellCheckCollatorTest, PreAnalyzedUpdateProcessorTest, HardAutoCommitTest, ZkNodePropsTest, AtomicUpdatesTest, DocValuesMultiTest, TestStressLucene, TestDistributedGrouping, TestRangeQuery, NoCacheHeaderTest, DateFieldTest, DocValuesMissingTest, CollectionsAPIDistributedZkTest, StatsComponentTest, DisMaxRequestHandlerTest, SuggesterWFSTTest, DefaultValueUpdateProcessorTest, CoreAdminRequestStatusTest, XsltUpdateRequestHandlerTest, TestOverriddenPrefixQueryForCustomFieldType, TestStressRecovery, OutputWriterTest, CoreAdminCreateDiscoverTest, HdfsBasicDistributedZkTest, TestFreeTextSuggestions, PluginInfoTest, MinimalSchemaTest, RequestHandlersTest, TestFastLRUCache, TestBinaryResponseWriter, RAMDirectoryFactoryTest, TestFieldTypeResource, TestFuzzyAnalyzedSuggestions, TestManagedSchema, TestCopyFieldCollectionResource, SuggestComponentTest, CachingDirectoryFactoryTest, SortByFunctionTest, PrimUtilsTest, TestNoOpRegenerator, TestSolrCoreProperties, TestMultiCoreConfBootstrap, NotRequiredUniqueKeyTest, DirectSolrConnectionTest, BasicDistributedZk2Test, QueryResultKeyTest, DirectUpdateHandlerTest, ConvertedLegacyTest, TriLevelCompositeIdRoutingTest, TestQuerySenderListener, TestCoreDiscovery, TestSolrXmlPersistor, TestCodecSupport, TestAnalyzeInfixSuggestions, TestBlendedInfixSuggestions, TestDefaultSimilarityFactory, NumericFieldsTest, TestAnalyzedSuggestions, TestSolrJ, DirectSolrSpellCheckerTest, ZkSolrClientTest, TestDistributedSearch, TestCSVResponseWriter, ConnectionManagerTest, TestRandomFaceting, SolrIndexConfigTest, TestIBSimilarityFactory, TestSearcherReuse, TestSolrDeletionPolicy2, TestFiltering, ChaosMonkeyNothingIsSafeTest, StandardRequestHandlerTest, TermVectorComponentDistributedTest, TestJmxIntegration, TestXIncludeConfig, TestDefaultSearchFieldResource, TestRecovery, TestSweetSpotSimilarityFactory, HighlighterConfigTest, HdfsUnloadDistributedZkTest, SimplePostToolTest, TestStandardQParsers, TestWordDelimiterFilterFactory, OpenCloseCoreStressTest, OverseerTest]
   [junit4] Completed on J1 in 59.50s, 9 tests, 1 error <<< FAILURES!

[...truncated 598 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux@2/lucene/common-build.xml:921: There were test failures: 393 suites, 1637 tests, 1 error, 38 ignored (17 assumptions)

Total time: 62 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b15 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message