Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Date: Tue, 29 Nov 2016 16:32:56 +0000 (UTC)
From: Sumit Nigam <sumit_only@yahoo.com.INVALID>
Reply-To: Sumit Nigam <sumit_only@yahoo.com>
To: Hbase-user <user@hbase.apache.org>
Message-ID: <2114650584.2702067.1480437176531@mail.yahoo.com>
Subject: Multi-homed hosts with Hbase
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_2702066_324378147.1480437176504"
References: <2114650584.2702067.1480437176531.ref@mail.yahoo.com>
archived-at: Tue, 29 Nov 2016 16:35:14 -0000

------=_Part_2702066_324378147.1480437176504
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

I am using multi-homed kerberized hosts with hbase 0.98.

The /etc/hosts files are uniformly containing FQDNs for cluster nodes. Howe=
ver, region server fails to register with master server with the error:

2016-11-17 15:05:47,700 WARN =C2=A0[regionserver54723] regionserver.HRegion=
Server: error telling master we are up

com.google.protobuf.ServiceException: java.io.IOException: Couldn't setup c=
onnection for INFACSDEVKERB/dvterhpdw02.devadmin.nbsdev.co.uk@DEVADMIN.NBSD=
EV.CO.UK to INFACSDEVKERB/dvterhpdw03.devadmin.nbsdev.co.uk@DEVADMIN.NBSDEV=
.CO.UK
 at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java=
:1679)
 at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.=
callBlockingMethod(RpcClient.java:1720)
 at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$Reg=
ionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusP=
rotos.java:8277)
 at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegio=
nServer.java:2190)
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused b=
y GSSException: No valid credentials provided (Mechanism level: Server not =
found in Kerberos database (7))]
 at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl=
ient.java:211)
 at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSa=
slRpcClient.java:179)

The error indicates some DNS lookup problem. So, my question is do I need t=
o enable some config in Hbase to allow multi-homed hosts? Also, I assume th=
at if /etc/hosts is setup correctly, then entries in DNS server should not =
matter given the order of lookup is /etc/hosts and then DNS in my setup. I =
looked at some documentation and bugs (https://issues.apache.org/jira/brows=
e/HBASE-12954 or=C2=A0https://issues.apache.org/jira/browse/HADOOP-8198) wh=
ich seem to indicate multi-home support is not straight forward?

Zookeeper is not managed by Hbase. As a next step, I will dump zookeeper da=
ta from hbase shell to see what hostnames/ fqdns are being reported by mast=
er and region server. Any other suggestions can I try?

Thanks,

Sumit
------=_Part_2702066_324378147.1480437176504--