Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Sat, 17 Jun 2017 02:07:00 +0000 (UTC)
From: "Josh Elser (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.13080492.1497640088000.38501.1497665220064@Atlassian.JIRA>
In-Reply-To: <JIRA.13080492.1497640088000@Atlassian.JIRA>
References: <JIRA.13080492.1497640088000@Atlassian.JIRA> <JIRA.13080492.1497640088174@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HBASE-18226) Disable reverse DNS lookup at
 HMaster and use default hostname provided by RegionServer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 17 Jun 2017 02:07:05 -0000


    [ https://issues.apache.org/jira/browse/HBASE-18226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052635#comment-16052635 ] 

Josh Elser commented on HBASE-18226:
------------------------------------

bq. We do not want HMaster to do reverse DNS lookup because HMaster VM's /etc/hosts does not have regionserver VM's FQDN mappings

bq. it uses FQDN names here but on HMaster side, it will do reverse DNS lookup which cannot be resolved.

bq. The patch simply stops HMaster doing reverse DNS lookup.

Without yet looking at the patch, this is a little worrisome. The rDNS lookup by the Master is to prevent RS's from joining the cluster which don't have a "stable" DNS. That is, if the Master cannot perform a DNS lookup on the IP of the RS that is reporting for duty and get the same hostname that the RS *thinks* it has, a client would also be very likely to get a different hostname.

Like Nick points out, when Kerberos is enabled, inconsistent DNS means the cluster is unusable. RPCs over SASL with GSSAPI(krb5) require that the instance component of the Kerberos principal match the hostname that a client used to connect to a server. For example, if a RS has a principal {{hbase/host1.domain.com}}, any RPC to that RS with a hostname *other than* {{host1.domain.com}} is guaranteed to fail with an authentication error.

This all leads me to wonder how the Master presently sends RPCs to RegionServers with Kerberos enabled on Azure. Maybe the master can only do forward DNS lookups and not reverse DNS lookups? I think that would be a plausible explanation.

Will try to take a look at the patch and give it some more thought.

> Disable reverse DNS lookup at HMaster and use default hostname provided by RegionServer
> ---------------------------------------------------------------------------------------
>
>                 Key: HBASE-18226
>                 URL: https://issues.apache.org/jira/browse/HBASE-18226
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Duo Xu
>         Attachments: HBASE-18226.001.patch
>
>
> This JIRA is to address the similar problem as HBASE-12954, but there are some little differences,
> 1. HBASE-12954 provides the configuration "hbase.regionserver.hostname" so users can configure it on every regionserver with preferred hostnames. However, this configuration cannot be set through Ambari or other configuration management tools because each regionserver has a different value of this setting, which means each node needs a different hbase-site.xml.
> 2. In Azure HDInsight clusters, we want to give each RegionServer/workernode a FQDN by modifying /etc/hosts on that node. We do not want HMaster to do reverse DNS lookup because HMaster VM's /etc/hosts does not have regionserver VM's FQDN mappings. In current implementation when regionserver starts, 
> {code}
> String hostName = shouldUseThisHostnameInstead() ? useThisHostnameInstead :
>       rpcServices.isa.getHostName();
> {code}
> it uses FQDN names here but on HMaster side, it will do reverse DNS lookup which cannot be resolved.
> {code}
>  // if regionserver passed hostname to use,
>  // then use it instead of doing a reverse DNS lookup
>  ServerName rs = master.getServerManager().regionServerStartup(request, ia);
> {code}
> My proposed fix is to add a new configuration "*hbase.regionserver.hostname.auto*". If it is set to true, then Regionserver will use the value returned by *rpcServices.isa.getHostName()* as the hostname overwriting whatever users specifies in "*hbase.regionserver.hostname*" and send to HMaster as part of RegionServerStartupRequest. HMaster will not do reverse DNS lookup, which has been implemented in HBASE-12954. If users want to provide their own hostnames in "*hbase.regionserver.hostname*", "*hbase.regionserver.hostname.auto*" must be false.
> I will submit a patch later today.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)