Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 3 Mar 2017 03:01:45 +0000 (UTC)
From: "stack (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13047588.1488439231000.50601.1488510105352@Atlassian.JIRA>
In-Reply-To: <JIRA.13047588.1488439231000@Atlassian.JIRA>
References: <JIRA.13047588.1488439231000@Atlassian.JIRA> <JIRA.13047588.1488439231905@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-17718) Difference between RS's servername
 and its ephemeral node cause SSH stop working
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 03 Mar 2017 03:01:51 -0000


    [ https://issues.apache.org/jira/browse/HBASE-17718?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D158=
93632#comment-15893632 ]=20

stack commented on HBASE-17718:
-------------------------------

#1 because HBASE-9593 is wrong (registering in zk with local name rather th=
an master-provided name BEFORE we've talked to the master to get what our c=
luster name is supposed to be).

There is also something odd about HBASE-9593 looking at it now again. A reg=
ionserver is being 'registered' via a reading of the zk data, not via heart=
beat so it must be something like a master that has joined a cluster that i=
s already up after a master crash. It looks like an extremely rare case whe=
re an ephemeral node has not evaporated yet and in the meantime a master cr=
ashes and then a backup master joins the cluster. Looks like we need a more=
 rigorous accounting of cluster servers when a backup master joins a runnin=
g cluster. It can read candidate servers by looking in zk but it should wai=
t on a heartbeat before adding a candidate regionserver to its cluster set.=
 We can open a new issue to do this so we prevent a version of HBASE-9593 a=
rising again post revert.

> Difference between RS's servername and its ephemeral node cause SSH stop =
working
> -------------------------------------------------------------------------=
-------
>
>                 Key: HBASE-17718
>                 URL: https://issues.apache.org/jira/browse/HBASE-17718
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 2.0.0, 1.2.4, 1.1.8
>            Reporter: Allan Yang
>            Assignee: Allan Yang
>
> After HBASE-9593, RS put up an ephemeral node in ZK before reporting for =
duty. But if the hosts config (/etc/hosts) is different between master and =
RS, RS's serverName can be different from the one stored the ephemeral zk n=
ode. The email metioned in HBASE-13753 (http://mail-archives.apache.org/mod=
_mbox/hbase-user/201505.mbox/%3CCANZDn9ueFEEuZMx=3DpZdmtLsdGLyZz=3Drrm1N6EQ=
vLswYc1z-H=3Dg@mail.gmail.com%3E) is exactly what happened in our productio=
n env.=20
> But what the email didn't point out is that the difference between server=
Name in RS and zk node can cause SSH stop to work. as we can see from the c=
ode in {{RegionServerTracker}}
> {code}
>   @Override
>   public void nodeDeleted(String path) {
>     if (path.startsWith(watcher.rsZNode)) {
>       String serverName =3D ZKUtil.getNodeName(path);
>       LOG.info("RegionServer ephemeral node deleted, processing expiratio=
n [" +
>         serverName + "]");
>       ServerName sn =3D ServerName.parseServerName(serverName);
>       if (!serverManager.isServerOnline(sn)) {
>         LOG.warn(serverName.toString() + " is not online or isn't known t=
o the master."+
>          "The latter could be caused by a DNS misconfiguration.");
>         return;
>       }
>       remove(sn);
>       this.serverManager.expireServer(sn);
>     }
>   }
> {code}
> The server will not be processed by SSH/ServerCrashProcedure. The regions=
 on this server will not been assigned again until master restart or failov=
er.
> I know HBASE-9593 was to fix the issue if RS report to duty and crashed b=
efore it can put up a zk node. It is a very rare case(And controllable=EF=
=BC=8C just fix the bug making rs to crash). But The issue I metioned can h=
appened more often(and uncontrollable, can't be fixed in HBase, due to DNS,=
 hosts config, etc.) and have more severe consequence.
> So here I offer some solutions to discuss:
> 1. Revert HBASE-9593 from all branches, Andrew Purtell has reverted it in=
 branch-0.98
> 2. Abort RS if master return a different name, otherwise SSH can't work p=
roperly
> 3. Master accepts whatever servername reported by RS and don't change it.
> 4.correct the zk node if master return another name( idea from Ted Yu)
> =20


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)