zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Sherman <bensher...@gmail.com>
Subject Tickling the election ports
Date Fri, 02 Jun 2017 21:46:59 GMT
Hi all,

Regarding my recent outages, I have a suspicion that there is some stateful
connection tracking happening between my servers that is invisible to me.
(In this case, it's across availability zones in AWS VPCs).

This has come up in both a JIRA ticket at
https://issues.apache.org/jira/browse/ZOOKEEPER-1748 and a PR in the git
repo at https://github.com/apache/zookeeper/pull/83

I believe that when an enseble is started that there are connections setup
between each server on port 3888 (among others). As the server is normally
healthy, there is no traffic across that connection beyond the initial
election. At some point with no traffic, the black box NAT device removes
it from the state table but does not send a FIN or RST down the pipe, but
the service thinks the connection still exists. During a failure, ZK will
attempt to send traffic down said pipe during a new election, but it won't
work, and will have to wait for the system timeouts to kill the connection.

Am I correct in the following assumptions:

1. When an ensemble is healthy, no traffic goes across the election ports.
2. There is no way to trigger traffic across those ports (four letter
command or otherwise) without causing a failure in the ensemble.
3. I can cause traffic on those ports across the entire ensemble should I
restart any node in the ensemble.

Finally, is there any way to shine any light on the above issues that
highlight this? I have considered forking 3.4.10 to do this, but the
overhead required is more than I can afford right now going down the line.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message