You're right in that node #3 declared itself leader given the sequence of notifications it received: it received one notification from node #1 saying that it was looking in the first few lines of your log excerpt and at least one from itself. However, node #3 does not have enough support (followers), so it won't exercise leadership. From the notifications, it sounds like node #1 is the one exercising leadership.

-Flavio
 
On Mar 22, 2011, at 8:11 AM, Jeremy Stribling wrote:

Thanks for the response.  I thought that the "my state" in this line,
printed in node #3's log:


2672 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
215 (n.leader), 12884902548 (n.zxid), 3 (n.round), FOLLOWING
(n.state), 126 (n.sid), LEADING (my state)

indicated that node #3 was the leader, but I'm probably misinterpreting
it (I haven't had a chance to look through the source yet to figure it
out for sure).  In any case, what I think are the relevant notifications
of node #1's logs look like this:



307122 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
37 (n.leader), 17179869831 (n.zxid), 1 (n.round), LOOKING (n.state),
37 (n.sid), LEADING (my state)
307142 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
37 (n.leader), 17179869831 (n.zxid), 3 (n.round), LOOKING (n.state),
37 (n.sid), LEADING (my state)
310850 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
215 (n.leader), 17179869918 (n.zxid), 4 (n.round), LOOKING (n.state),
215 (n.sid), LOOKING (my state)
310850 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
37 (n.leader), 17179869831 (n.zxid), 3 (n.round), LEADING (n.state),
37 (n.sid), LOOKING (my state)
311051 [QuorumPeer:/0.0.0.0:2888] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification
time out: 400
311053 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
37 (n.leader), 17179869831 (n.zxid), 3 (n.round), LEADING (n.state),
37 (n.sid), LOOKING (my state)
311054 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
215 (n.leader), 17179869918 (n.zxid), 4 (n.round), LOOKING (n.state),
215 (n.sid), LOOKING (my state)
311454 [QuorumPeer:/0.0.0.0:2888] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification
time out: 800
311456 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
37 (n.leader), 17179869831 (n.zxid), 3 (n.round), LEADING (n.state),
37 (n.sid), LOOKING (my state)
311457 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
215 (n.leader), 17179869918 (n.zxid), 4 (n.round), LOOKING (n.state),
215 (n.sid), LOOKING (my state)
312257 [QuorumPeer:/0.0.0.0:2888] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification
time out: 1600
312260 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
215 (n.leader), 17179869918 (n.zxid), 4 (n.round), LOOKING (n.state),
215 (n.sid), LOOKING (my state)
312263 [WorkerReceiver Thread] INFO
org.apache.zookeeper.server.quorum.FastLeaderElection  - Notification:
37 (n.leader), 17179869831 (n.zxid), 3 (n.round), LEADING (n.state),
37 (n.sid), LOOKING (my state)

which, according to my earlier logic, seems to indicate that node #1
never even thought it was following node #3.

Anyway, I will put the logs together and make a JIRA tomorrow if I get
some time, and will follow up here with a link.  Thanks again,

Jeremy



flavio
junqueira
 
research scientist
 
fpj@yahoo-inc.com
direct +34 93-183-8828
 
avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301