Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 4 Mar 2017 17:43:45 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13048257.1488609764000.63246.1488649425261@Atlassian.JIRA>
In-Reply-To: <JIRA.13048257.1488609764000@Atlassian.JIRA>
References: <JIRA.13048257.1488609764000@Atlassian.JIRA> <JIRA.13048257.1488609764523@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17733) Undo registering regionservers in
 zk with ephemeral nodes; its more trouble than its worth
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 04 Mar 2017 17:43:51 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895790#comment-15895790 ] 

stack commented on HBASE-17733:
-------------------------------

[~busbey] For example, why not just solve this problem by having the Master watch for new ephemeral znodes and only add the RS to its internals at that point? Just because we need to ask the master what name it sees for us doesn't mean we have to use that request as the time to consider the RS fully bootstrapped.

I considered versions of the above but thought there too much risk involved changing order of registration afraid I'd break something unforeseen and queasy at the thought of doubling-down on two daemons having to go via a third service to establish a connectivity they've already proven. If a refactor/redo, I wanted to do it 'right' -- hence this issue.

That said, this simple 'offset' registration suggestion of yours I did not consider. It is a better idea than any I had. I took a look.

In Master, we have RegionServerTracker. It already takes care of setting watchers looking for changes in the zk RS znode. Currently whenever a  change, we refresh our list of online servers without regard. I could do a parse and if the server is a 'new' entry not seen before, call a new version of ServerManager#regionServerStartup (would have to make sure watcher triggers properly in all cases). So far not bad.

But before we register a server, we check its current timestamp to check for clockskew (Heartbeat includes the current RS ts. Ephemeral znode is stamped once with time of its stamping). If the skew is unacceptable, we throw a ClockOutOfSyncException. This goes back as the result of the RS's first check in ('reportForDuty') and the RS will shut itself down. We'd have to build up a replacement channel for this clock skew check if we register new servers via changed child watcher (Just noticed: our clock skew check is poor; it is only done on first server registration -- we should check on every heartbeat).

So, it looks like your suggestion involves a good bit of work and risk. I'd rather spend the effort on removing zk from equation. WDYT

> Undo registering regionservers in zk with ephemeral nodes; its more trouble than its worth
> ------------------------------------------------------------------------------------------
>
>                 Key: HBASE-17733
>                 URL: https://issues.apache.org/jira/browse/HBASE-17733
>             Project: HBase
>          Issue Type: Brainstorming
>            Reporter: stack
>
> Elsewhere, we are undoing the use of ZK (replication current WAL offset, regions-in-transition, etc).
> I have another case where using ZK, while convenient (call-backs), has holes.
> The scenario is prompted by review of HBASE-9593.
> Currently, a RS registers with the Master by calling the Master's reportForDuty. After the Master responds with the name we are to use for ourselves (as well as other properties we need to 'run'), we then turnaround and do a new RPC out to the zk ensemble to register an ephemeral znode for the RS.
> We notice a RS has gone away -- crashed -- because its znode evaporates and the Master has a watcher triggered notifying it the RS has gone (after a zk session timeout of tens of seconds).  Cumbersome (Setting watchers, zk session timeouts) and indirect. Master then trips the server shutdown handler which does reassign of regions from the crashed server.
> In HBASE-9593, we were trying to handle the rare but possible case where the RS would die after registering w/ the Master but before we put up our ephemeral znode. In this case a RS would live in the Master's internals forever because there is no ephemeral znode to expire to do cleanup and removal of the never-started RS.
> Lets get ZK out of the loop. Then only the Master and RS involved heartbeating each other.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)