lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1493 - Still Failing!
Date Mon, 14 Apr 2014 13:48:33 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1493/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([DF8480E0174E5202:DB8C0F1305EBBD23]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
	... 47 more




Build Log:
[...truncated 10972 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-DF8480E0174E5202-001/init-core-data-001
   [junit4]   2> 2300946 T7064 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2300948 T7064 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 2300948 T7064 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2300959 T7064 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 2300960 T7064 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2300961 T7065 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2301062 T7064 oasc.ZkTestServer.run start zk server on port:51243
   [junit4]   2> 2301064 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2301080 T7071 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15fd7a15 name:ZooKeeperConnection Watcher:127.0.0.1:51243 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2301080 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2301097 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2301100 T7073 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a39defc name:ZooKeeperConnection Watcher:127.0.0.1:51243 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2301100 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2301101 T7064 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2301125 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2301130 T7075 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e78ce32 name:ZooKeeperConnection Watcher:127.0.0.1:51243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2301137 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2301138 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2301164 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2301166 T7077 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79f88da6 name:ZooKeeperConnection Watcher:127.0.0.1:51243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2301168 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2301180 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2301203 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2301269 T7064 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51243_solr
   [junit4]   2> 2301273 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2301309 T7064 oasc.Overseer.start Overseer (id=91585341479190531-127.0.0.1:51243_solr-n_0000000000) starting
   [junit4]   2> 2301346 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2301408 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2301456 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2301518 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2301583 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2301651 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2301771 T7079 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2301777 T7064 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2301783 T7079 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51243_solr
   [junit4]   2> 2301784 T7078 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2301790 T7064 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2301795 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2301801 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2301804 T7081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f07d3e1 name:ZooKeeperConnection Watcher:127.0.0.1:51243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2301805 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2301807 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2301813 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 2301820 T7075 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2301821 T7081 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2301825 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2301830 T7078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2301832 T7078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2301832 T7078 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2301832 T7078 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2301840 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2301842 T7075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2301842 T7081 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2302326 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2302356 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2302376 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2302381 T7078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2302382 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2302393 T7078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2302394 T7078 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2302394 T7078 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2302402 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2302509 T7075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2302510 T7081 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2302884 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2302915 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2302928 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2302929 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2302929 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2302932 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2302933 T7078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2302943 T7078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2302944 T7078 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2302944 T7078 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2302950 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2303059 T7081 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2303059 T7075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2303434 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2303455 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2303471 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2303471 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2303471 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2303474 T7078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2303476 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2303487 T7078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2303487 T7078 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2303488 T7078 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2303494 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2303602 T7081 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2303603 T7075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2303978 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2304022 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304022 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304022 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304027 T7078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2304028 T7078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2304029 T7078 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2304029 T7078 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2304035 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304141 T7075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2304141 T7081 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2304524 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2304572 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304573 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304573 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304578 T7078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2304579 T7078 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2304579 T7078 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2304580 T7078 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2304591 T7077 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2304699 T7075 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2304699 T7081 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2305075 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2305096 T7081 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2305101 T7064 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51243 51243
   [junit4]   2> 2305132 T7065 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51243 51243
   [junit4]   2> 2305357 T7064 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   2> 2305357 T7078 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51243_solr
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2305360 T7082 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2305368 T7064 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
   [junit4]   2> 2305369 T7064 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2305370 T7083 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2305470 T7064 oasc.ZkTestServer.run start zk server on port:51249
   [junit4]   2> 2305472 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2305479 T7089 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7092aec name:ZooKeeperConnection Watcher:127.0.0.1:51249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2305479 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2305482 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2305492 T7091 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c3122de name:ZooKeeperConnection Watcher:127.0.0.1:51249 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2305492 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2305498 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2305502 T7093 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@759d8ab name:ZooKeeperConnection Watcher:127.0.0.1:51249 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2305503 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2305503 T7064 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2305511 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2305520 T7064 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2305529 T7064 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2305535 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2305543 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2305547 T7095 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e109f2a name:ZooKeeperConnection Watcher:127.0.0.1:51249/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2305548 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2305549 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2305555 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2305567 T7064 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51249_solr
   [junit4]   2> 2305567 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2305572 T7064 oasc.Overseer.start Overseer (id=91585341768269827-127.0.0.1:51249_solr-n_0000000000) starting
   [junit4]   2> 2305577 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2305586 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2305594 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2305604 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2305613 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2305624 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2305643 T7097 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2305647 T7097 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51249_solr
   [junit4]   2> 2305647 T7096 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2305649 T7064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2305654 T7096 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2305655 T7096 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2305656 T7096 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2305665 T7095 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2305667 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2305704 T7064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2305709 T7095 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2305713 T7096 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2305715 T7096 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 2305715 T7096 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 2305716 T7096 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2305723 T7095 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2305831 T7089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2305868 T7064 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51249 51249
   [junit4]   2> 2305891 T7083 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51249 51249
   [junit4]   2> 2306107 T7064 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   2> 2306108 T7096 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51249_solr
   [junit4]   2> 2306109 T7098 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2306127 T7064 oas.SolrTestCaseJ4.setUp ###Starting testBadQueueItem
   [junit4]   2> 2306129 T7064 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2306131 T7099 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2306231 T7064 oasc.ZkTestServer.run start zk server on port:51254
   [junit4]   2> 2306233 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2306239 T7105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4dbbba8b name:ZooKeeperConnection Watcher:127.0.0.1:51254 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2306240 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2306245 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2306250 T7107 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28c1c2d7 name:ZooKeeperConnection Watcher:127.0.0.1:51254 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2306251 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2306251 T7064 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2306261 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2306264 T7109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33d6b622 name:ZooKeeperConnection Watcher:127.0.0.1:51254/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2306265 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2306265 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2306270 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2306274 T7111 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@770aec48 name:ZooKeeperConnection Watcher:127.0.0.1:51254/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2306274 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2306276 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2306282 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2306294 T7064 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51254_solr
   [junit4]   2> 2306295 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2306302 T7064 oasc.Overseer.start Overseer (id=91585341818142723-127.0.0.1:51254_solr-n_0000000000) starting
   [junit4]   2> 2306308 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2306320 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2306330 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2306340 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2306349 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2306361 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2306379 T7113 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2306381 T7064 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2306381 T7113 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51254_solr
   [junit4]   2> 2306382 T7112 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2306388 T7064 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2306393 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2306402 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2306405 T7115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f241394 name:ZooKeeperConnection Watcher:127.0.0.1:51254/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2306406 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2306409 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2306417 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 2306424 T7109 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2306424 T7115 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2306430 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2306434 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2306435 T7112 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2306435 T7112 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2306436 T7112 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2306442 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2306444 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2306444 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2306599 T7081 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2306600 T7081 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2306601 T7081 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2306931 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2306957 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2306978 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2306987 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2306988 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2307001 T7112 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2307001 T7112 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2307001 T7112 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2307009 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2307121 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2307122 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2307491 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2307517 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2307533 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2307533 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2307533 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2307537 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2307538 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2307548 T7112 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2307548 T7112 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2307548 T7112 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2307554 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2307664 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2307664 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2308039 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2308065 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2308082 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308082 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308083 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308087 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2308096 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308205 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2308205 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2308244 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308244 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308244 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308253 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308254 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2308256 T7112 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core0",
   [junit4]   2> 	  "core_node_name":"node0",
   [junit4]   2> 	  "collection":"",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2308266 T7112 oasc.Overseer$ClusterStateUpdater.checkKeyExistence ERROR Skipping invalid Overseer message because it has no collection specified: {
   [junit4]   2> 	  "operation":"deletecore",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core0",
   [junit4]   2> 	  "core_node_name":"node0",
   [junit4]   2> 	  "collection":""}
   [junit4]   2> 2308274 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308277 T7112 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2308277 T7112 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection2 with shards [shard1, shard2, shard3]
   [junit4]   2> 2308277 T7112 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2308284 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308394 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2308394 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2308760 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard2/election
   [junit4]   2> 2308792 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard2
   [junit4]   2> 2308807 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308807 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308807 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308810 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2308811 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308820 T7112 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2308820 T7112 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2308820 T7112 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2308826 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2308934 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2308934 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2309312 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard1/election
   [junit4]   2> 2309336 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard1
   [junit4]   2> 2309351 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309351 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309351 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309355 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2309356 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309365 T7112 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection2",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 2309365 T7112 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2309366 T7112 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2309371 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309479 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2309479 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2309858 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection2/leader_elect/shard3/election
   [junit4]   2> 2309890 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection2/leaders/shard3
   [junit4]   2> 2309905 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309905 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309905 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2309909 T7112 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2309915 T7111 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2310023 T7115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2310024 T7109 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2310070 T7115 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2310079 T7064 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51254 51254
   [junit4]   2> 2310096 T7099 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51254 51254
   [junit4]   2> 2310324 T7064 oas.SolrTestCaseJ4.tearDown ###Ending testBadQueueItem
   [junit4]   2> 2310326 T7112 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51254_solr
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2310329 T7116 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2310336 T7064 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
   [junit4]   2> 2310338 T7064 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2310339 T7117 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2310440 T7064 oasc.ZkTestServer.run start zk server on port:51260
   [junit4]   2> 2310444 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2310457 T7123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7af9f405 name:ZooKeeperConnection Watcher:127.0.0.1:51260/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2310458 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2310462 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2310469 T7125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e246ec0 name:ZooKeeperConnection Watcher:127.0.0.1:51260 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2310470 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2310476 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2310479 T7127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29ce5e67 name:ZooKeeperConnection Watcher:127.0.0.1:51260 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2310479 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2310479 T7064 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2310491 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2310497 T7064 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2310502 T7064 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2310506 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2310513 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2310516 T7129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63d774d1 name:ZooKeeperConnection Watcher:127.0.0.1:51260/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2310516 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2310519 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2310525 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2310532 T7123 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2310532 T7129 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2310534 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2310537 T7131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5102c569 name:ZooKeeperConnection Watcher:127.0.0.1:51260/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2310538 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2310539 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2310545 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2310556 T7064 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51260_solr
   [junit4]   2> 2310556 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2310562 T7064 oasc.Overseer.start Overseer (id=91585342093852676-127.0.0.1:51260_solr-n_0000000000) starting
   [junit4]   2> 2310567 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2310577 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2310588 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2310601 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2310610 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2310622 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2310641 T7133 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2310644 T7133 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51260_solr
   [junit4]   2> 2310646 T7132 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2310652 T7132 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2310655 T7132 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2310655 T7132 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 2310655 T7132 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2310664 T7131 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2310667 T7123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2310668 T7129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2311150 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2311189 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2311204 T7131 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2311206 T7064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2311208 T7132 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2311211 T7123 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2311212 T7129 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2311216 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2311220 T7135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35eeb6f1 name:ZooKeeperConnection Watcher:127.0.0.1:51260/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2311220 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2311220 T7131 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2311223 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2311229 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2311236 T7123 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2311237 T7135 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2311241 T7131 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2311241 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2311248 T7132 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 2311248 T7132 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 2311255 T7131 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2311267 T7064 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2311279 T7131 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2311287 T7131 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2311393 T7123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2311393 T7135 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2311395 T7064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2311401 T7123 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2311401 T7135 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 2311406 T7064 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51260 51260
   [junit4]   2> 2311416 T7117 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51260 51260
   [junit4]   2> 2311706 T7064 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 2311710 T7132 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51260_solr
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2311713 T7136 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2311724 T7064 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 2311725 T7064 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2311727 T7137 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2311828 T7064 oasc.ZkTestServer.run start zk server on port:51267
   [junit4]   2> 2311831 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2311843 T7143 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b5201f8 name:ZooKeeperConnection Watcher:127.0.0.1:51267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2311844 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2311846 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2311856 T7145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40f93907 name:ZooKeeperConnection Watcher:127.0.0.1:51267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2311857 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2311868 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2311875 T7147 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a7a1945 name:ZooKeeperConnection Watcher:127.0.0.1:51267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2311876 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2311876 T7064 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2311888 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2311896 T7064 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2311902 T7064 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2311907 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2311920 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2311934 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2311938 T7149 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cf11273 name:ZooKeeperConnection Watcher:127.0.0.1:51267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2311938 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2311941 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2311947 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2311961 T7064 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51267_solr
   [junit4]   2> 2311961 T7064 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2311969 T7064 oasc.Overseer.start Overseer (id=91585342185209859-127.0.0.1:51267_solr-n_0000000000) starting
   [junit4]   2> 2311976 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2311992 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2312005 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2312017 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2312030 T7064 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2312051 T7151 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2312054 T7151 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51267_solr
   [junit4]   2> 2312060 T7150 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2312061 T7150 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 2312063 T7150 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2312066 T7143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2312077 T7150 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2312080 T7143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2312088 T7150 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2312095 T7150 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2312097 T7150 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 2312105 T7149 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2312107 T7143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 2312166 T7064 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51267 51267
   [junit4]   2> 2312177 T7137 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51267 51267
   [junit4]   2> 2312405 T7064 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   2> 2312408 T7150 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:51267_solr
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2312413 T7152 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   2> 2312420 T7064 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 2312421 T7064 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2312422 T7153 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2312524 T7064 oasc.ZkTestServer.run start zk server on port:51272
   [junit4]   2> 2312525 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312532 T7159 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c5f7fdf name:ZooKeeperConnection Watcher:127.0.0.1:51272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312533 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312535 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312541 T7161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51f797e1 name:ZooKeeperConnection Watcher:127.0.0.1:51272 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312542 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312548 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312554 T7163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@359969dc name:ZooKeeperConnection Watcher:127.0.0.1:51272 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312555 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312555 T7064 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2312564 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2312576 T7064 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312578 T7164 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312584 T7166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52cd6135 name:ZooKeeperConnection Watcher:127.0.0.1:51272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312584 T7064 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312586 T7168 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29a9090b name:ZooKeeperConnection Watcher:127.0.0.1:51272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312586 T7164 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312587 T7064 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2312588 T7164 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2312595 T7064 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2312596 T7164 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2312601 T7064 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2312609 T7164 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51272_solr
   [junit4]   2> 2312609 T7164 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2312610 T7064 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 2312618 T7164 oasc.Overseer.start Overseer (id=91585342230429700-127.0.0.1:51272_solr-n_0000000000) starting
   [junit4]   2> 2312620 T7166 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 2312623 T7064 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2312627 T7164 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2312636 T7164 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2312650 T7164 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2312661 T7164 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2312674 T7164 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2312689 T7164 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2312713 T7164 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 2312715 T7170 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2312721 T7164 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312725 T7172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ecd7dd9 name:ZooKeeperConnection Watcher:127.0.0.1:51272/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312725 T7164 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312726 T7170 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=91585342230429700-127.0.0.1:51272_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 2312726 T7170 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51272_solr
   [junit4]   2> 2312726 T7170 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:294)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:207)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 2312736 T7164 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51272_solr
   [junit4]   2> 2312736 T7164 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2312742 T7164 oasc.Overseer.start Overseer (id=91585342230429701-127.0.0.1:51272_solr-n_0000000001) starting
   [junit4]   2> 2312778 T7174 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2312779 T7174 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes

[...truncated too long message...]

n.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=DF8480E0174E5202 -Dtests.slow=true -Dtests.locale=ar_DZ -Dtests.timezone=Europe/Mariehamn -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   18.8s | OverseerTest.testOverseerFailure <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([DF8480E0174E5202:DB8C0F1305EBBD23]:0)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
   [junit4]    > 	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
   [junit4]    > 	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
   [junit4]    > 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]    > 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
   [junit4]    > 	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]    > 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
   [junit4]    > 	... 47 more
   [junit4]   2> 2353741 T7064 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 2353742 T7064 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 2356745 T7064 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-DF8480E0174E5202-001
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_DZ, timezone=Europe/Mariehamn
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=1,free=80463088,total=299950080
   [junit4]   2> NOTE: All tests run in this JVM: [PeerSyncTest, TestBlendedInfixSuggestions, TestClassNameShortening, TestQuerySenderNoQuery, NoCacheHeaderTest, SolrCmdDistributorTest, SuggestComponentTest, TestBadConfig, TestSimpleQParserPlugin, CacheHeaderTest, TestFastOutputStream, TestHighlightDedupGrouping, TestDistributedMissingSort, TestDocBasedVersionConstraints, TestSolrXml, TestTrie, ZkNodePropsTest, TestFastWriter, ClusterStateUpdateTest, MBeansHandlerTest, TestHashPartitioner, SchemaVersionSpecificBehaviorTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, ShardSplitTest, TestRequestStatusCollectionAPI, TestGroupingSearch, TestSolrQueryParser, CoreAdminHandlerTest, ConvertedLegacyTest, TestInfoStreamLogging, DefaultValueUpdateProcessorTest, SpatialFilterTest, TestDefaultSimilarityFactory, TestDistribDocBasedVersion, TestSuggestSpellingConverter, TestManagedSchemaFieldResource, SuggesterTSTTest, OpenExchangeRatesOrgProviderTest, TestRecoveryHdfs, TestSystemIdResolver, FullSolrCloudDistribCmdsTest, TestReload, AnalysisAfterCoreReloadTest, ModifyConfFileTest, TestCloudManagedSchemaAddField, TestRangeQuery, TestPhraseSuggestions, SyncSliceTest, FieldFacetExtrasTest, ZkCLITest, TestSolr4Spatial, OverseerCollectionProcessorTest, TestNonDefinedSimilarityFactory, BadCopyFieldTest, HdfsDirectoryTest, RequiredFieldsTest, TestFieldResource, SolrCoreCheckLockOnStartupTest, TestModifyConfFiles, SignatureUpdateProcessorFactoryTest, TestCollectionAPI, DistributedSpellCheckComponentTest, QueryParsingTest, HdfsChaosMonkeySafeLeaderTest, TestCoreDiscovery, SoftAutoCommitTest, TestManagedSchema, StressHdfsTest, TestInitQParser, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, DirectSolrConnectionTest, TestSchemaSimilarityResource, TestPHPSerializedResponseWriter, SolrTestCaseJ4Test, SimplePostToolTest, DebugComponentTest, ReturnFieldsTest, SpellPossibilityIteratorTest, TestSerializedLuceneMatchVersion, HdfsBasicDistributedZk2Test, TestDistributedGrouping, TestFreeTextSuggestions, TestExpandComponent, TestPartialUpdateDeduplication, PingRequestHandlerTest, DeleteInactiveReplicaTest, TestCursorMarkWithoutUniqueKey, TestBinaryResponseWriter, TestUpdate, DistributedSuggestComponentTest, URLClassifyProcessorTest, TestCSVResponseWriter, TestRealTimeGet, TestJmxIntegration, CoreAdminRequestStatusTest, DeleteShardTest, RangeFacetTest, TestReloadAndDeleteDocs, TestExtendedDismaxParser, DirectUpdateHandlerTest, TestSchemaVersionResource, DirectSolrSpellCheckerTest, DocExpirationUpdateProcessorFactoryTest, LeaderElectionTest, ResourceLoaderTest, QueryFacetTest, TestCollationFieldDocValues, TestWordDelimiterFilterFactory, TestMaxScoreQueryParser, TestMergePolicyConfig, TestStandardQParsers, TestSolrXmlPersistence, SearchHandlerTest, TestElisionMultitermQuery, DirectUpdateHandlerOptimizeTest, UpdateParamsTest, TestFaceting, TestSolrCoreProperties, OverseerRolesTest, DateMathParserTest, TestCSVLoader, DistributedQueryComponentCustomSortTest, DocValuesMissingTest, CoreAdminCreateDiscoverTest, LeaderElectionIntegrationTest, BasicDistributedZkTest, TestReversedWildcardFilterFactory, AbstractAnalyticsFacetTest, TestOverriddenPrefixQueryForCustomFieldType, ParsingFieldUpdateProcessorsTest, TestArbitraryIndexDir, TestFastLRUCache, TestCloudManagedSchema, TestFieldTypeCollectionResource, ZkControllerTest, TestBinaryField, SolrPluginUtilsTest, RecoveryZkTest, BadIndexSchemaTest, TestFileDictionaryLookup, TestCopyFieldCollectionResource, NotRequiredUniqueKeyTest, AnalysisErrorHandlingTest, CircularListTest, SortByFunctionTest, DocValuesTest, HardAutoCommitTest, FieldAnalysisRequestHandlerTest, TestNonNRTOpen, HdfsUnloadDistributedZkTest, TestSolrIndexConfig, TestComplexPhraseQParserPlugin, RemoteQueryErrorTest, TestQuerySenderListener, SSLMigrationTest, TestComponentsName, TermsComponentTest, SolrXmlInZkTest, BasicDistributedZk2Test, TestLRUCache, TestImplicitCoreProperties, IndexSchemaTest, FastVectorHighlighterTest, DistributedTermsComponentTest, BadComponentTest, TestLMDirichletSimilarityFactory, FileBasedSpellCheckerTest, DistributedExpandComponentTest, NoFacetTest, TestFieldTypeResource, TestRecovery, TestRTGBase, XmlUpdateRequestHandlerTest, UniqFieldsUpdateProcessorFactoryTest, CachingDirectoryFactoryTest, CoreMergeIndexesAdminHandlerTest, TestIndexingPerformance, ScriptEngineTest, InfoHandlerTest, DistanceFunctionTest, ConnectionManagerTest, TestSolrDeletionPolicy2, PrimitiveFieldTypeTest, AsyncMigrateRouteKeyTest, TermVectorComponentTest, TestCollapseQParserPlugin, TestStressLucene, SolrIndexConfigTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest]
   [junit4] Completed in 55.85s, 9 tests, 1 error <<< FAILURES!

[...truncated 599 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1281: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:907: There were test failures: 396 suites, 1662 tests, 1 error, 42 ignored (15 assumptions)

Total time: 111 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message