hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhangduo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12953) RegionServer is not functionally working with AysncRpcClient in secure mode
Date Fri, 20 Feb 2015 15:14:13 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14329035#comment-14329035
] 

zhangduo commented on HBASE-12953:
----------------------------------

This is found in a failed run, https://builds.apache.org/job/PreCommit-HBASE-Build/12918/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
{noformat}
2015-02-20 01:44:17,961 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=47118] visibility.DefaultVisibilityLabelServiceImpl(224):
Adding the label TEST_VISIBILITY
2015-02-20 01:44:17,966 INFO  [main] client.ConnectionManager$HConnectionImplementation(1661):
Closing zookeeper sessionid=0x14ba4a6e11b015f
2015-02-20 01:44:17,967 DEBUG [main] ipc.AsyncRpcClient(244): Stopping async HBase RPC client
{noformat}

And this is in a success run, https://builds.apache.org/job/PreCommit-HBASE-Build/12926/artifact/hbase-shell/target/surefire-reports/org.apache.hadoop.hbase.client.TestShell-output.txt
{noformat}
2015-02-20 14:12:48,681 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=43394] visibility.DefaultVisibilityLabelServiceImpl(224):
Adding the label TEST_VISIBILITY
2015-02-20 14:12:48,684 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(381): master:57132-0x14ba75420fb0000,
quorum=localhost:63493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/visibility/labels
2015-02-20 14:12:48,684 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(381): regionserver:43394-0x14ba75420fb0001,
quorum=localhost:63493, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected,
path=/hbase/visibility/labels
2015-02-20 14:12:48,686 INFO  [main] client.ConnectionManager$HConnectionImplementation(1661):
Closing zookeeper sessionid=0x14ba75420fb015f
2015-02-20 14:12:48,688 DEBUG [main] ipc.AsyncRpcClient(244): Stopping async HBase RPC client
{noformat}

The failed one missing the zk watcher logs.

I'm not familiar with Visibility Labels. I skimmed the code in DefaultVisibilityLabelServiceImpl,
seems we do not update labelsCache directly when addLabels and just write it to zk and then
let a zk watcher notify us to update labelsCache? So there could be situation that after addLabels
returns successfully, the labelsCache still does not contain the added labels? And setAuth
will check labelsCache so there could be a cache miss?

[~stack], is there anybody can give some suggestions? Thanks.

> RegionServer is not functionally working with AysncRpcClient in secure mode
> ---------------------------------------------------------------------------
>
>                 Key: HBASE-12953
>                 URL: https://issues.apache.org/jira/browse/HBASE-12953
>             Project: HBase
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.0.0, 1.1.0
>            Reporter: Ashish Singhi
>            Assignee: zhangduo
>            Priority: Critical
>             Fix For: 2.0.0, 1.1.0
>
>         Attachments: HBASE-12953.patch, HBASE-12953_1.patch, HBASE-12953_2.patch, HBASE-12953_2.patch,
HBASE-12953_2.patch, HBASE-12953_2.patch, HBASE-12953_2.patch, HBASE-12953_3 (2).patch, HBASE-12953_3
(2).patch, HBASE-12953_3.patch, HBASE-12953_3.patch, HBASE-12953_3.patch, testcase.patch
>
>
> HBase version 2.0.0
> Default value for {{hbase.rpc.client.impl}} is set to AsyncRpcClient.
> When trying to install HBase with Kerberos, RegionServer is not working functionally.
> The following log is logged in its log file
> {noformat}
> 2015-02-02 14:59:05,407 WARN  [AsyncRpcChannel-pool1-t1] channel.DefaultChannelPipeline:
An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually
means the last handler in the pipeline did not handle the exception.
> io.netty.channel.ChannelPipelineException: org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded()
has thrown an exception; removed.
> 	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:499)
> 	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded(DefaultChannelPipeline.java:481)
> 	at io.netty.channel.DefaultChannelPipeline.addFirst0(DefaultChannelPipeline.java:114)
> 	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:97)
> 	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:235)
> 	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:214)
> 	at org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:194)
> 	at org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:157)
> 	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> 	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> 	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> 	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> 	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> 	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:253)
> 	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:288)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> 	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> 	at org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded(SaslClientHandler.java:154)
> 	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:486)
> 	... 20 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
> 	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> 	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> 	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> 	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> 	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> 	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> 	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
> {noformat}
> When set hbase.rpc.client.impl to RpcClientImpl, there seems to be no issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message