hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Murumkar <manoj.murum...@gmail.com>
Subject Re: Kerberos exception
Date Sun, 15 Mar 2015 17:04:06 GMT
Attached region server log again.

Ted,

Corresponding log for master is:

java.io.IOException: Couldn't setup connection for hbase/
sfdvgctmn001.gid.gap.com@SFDVGCT.COM to hbase/sfdvgctsn001.gid.gap.com@S

FDVGCT.COM

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)

        at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)

        at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)

        at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)

        at
org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)

        at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)

        at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)

        at
org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)

        at
org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)

        at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by:
org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
GSS initiate failed

        at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)

        at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

        at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)

On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> bq. Attached both files
>
> I only found hbase-site.xml in the attachment.
>
> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <manoj.murumkar@gmail.com>
> wrote:
>
> > Attached both files.
> >
> >
> >
> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <olorinbant@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> Traces (especially one for region server) look a bit incomplete, did
> >> you copy them fully?
> >>
> >> Also may help if you post relevant pieces of hbase-site.xml (with
> >> security configs).
> >>
> >> Thanks,
> >> Mikhail
> >>
> >>
> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
> >> <manoj.murumkar@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > We have a secured cluster. All components are working well, except
> >> hbase.
> >> > Specifically, this is what I see on regionserver:
> >> >
> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Kerberos principal name is hbase/
> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
> >> >
> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >> >
> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: Have read input token of size 627 for processing by
> >> > saslServer.evaluateResponse()
> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> >> > reading:GSS initiate fail
> >> >
> >> >
> >> > and on the master:
> >> >
> >> >
> >> > java.io.IOException: Couldn't setup connection for hbase/sfdvgctmn004.
> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
> >> >
> >> >         at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >> >
> >> >         at java.security.AccessController.doPrivileged(Native Method)
> >> >
> >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> >> >
> >> >         at
> >> >
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >> >
> >> >         at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >> >
> >> >         at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >> >
> >> >         at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >> >
> >> >         at
> >> >
> >>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >> >
> >> > All the keytabs are set properly. Has anyone seen this before?
> >> Appreciate
> >> > the time.
> >> >
> >> > Manoj
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Michael Antonov
> >>
> >
> >
>

Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message