hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ashish singhi <ashish.sin...@huawei.com>
Subject RE: HBase replication seems to be not working with Kerberos cross realm trust
Date Thu, 17 Dec 2015 07:03:45 GMT
Hi all.

After looking more into the code we found that currently cross realm trust can work in HBase
only when FQDN in the Kerberos principal for hbase processes is hostname.
So we changed the Kerberos principal accordingly and hbase replication is working fine.

May be we can enhance our Sasl framework to support non-hostname also as FQDN in the Kerberos
principal.

Regards,
Ashish Singhi

From: ashish singhi
Sent: 14 December 2015 19:03
To: user
Subject: HBase replication seems to be not working with Kerberos cross realm trust

Hi all.

We are using HBase 1.0.2 and Java 1.8.0_51
HBase replication is not working for us in Kerberos cross realm trust.
We have followed all the instructions provided at http://www.cloudera.com/content/www/en-us/documentation/archive/cdh/4-x/4-5-0/CDH4-Security-Guide/cdh4sg_topic_8_4.html

We are getting the following exception in the active cluster RS log,

2015-12-14 17:16:43,768 | WARN  | regionserver/host-10-19-92-192/10.19.92.192:21302.replicationSource,peer1
| Can't replicate because of a local or network error:  | org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:295)
java.io.IOException: Couldn't setup connection for hbase/hadoop.hadoop.com@HADOOP.COM<mailto:hbase/hadoop.hadoop.com@HADOOP.COM>
to hbase/hadoop.hadoop.com@HADOOP.COM<mailto:hbase/hadoop.hadoop.com@HADOOP.COM>
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:664)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:636)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:744)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:895)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:864)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1209)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
        at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.replicateWALEntry(AdminProtos.java:25690)
        at org.apache.hadoop.hbase.protobuf.ReplicationProtbufUtil.replicateWALEntry(ReplicationProtbufUtil.java:79)
        at org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:351)
        at org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint$Replicator.call(HBaseInterClusterReplicationEndpoint.java:335)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate
failed
        at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:153)
        at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:189)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:610)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:736)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:733)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
        at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:733)
        ... 13 more


Any pointers will be very helpful here.

P.S: We have tested Hadoop distcp tool and it seems to be working for us in the same env.

Regards,
Ashish Singhi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message