hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Antonov <olorinb...@gmail.com>
Subject Re: Kerberos exception
Date Mon, 16 Mar 2015 22:43:30 GMT
Glad it helped!

In general though, in this cases, if you turn on java kerberos debug
logging (via something like export
HADOOP_OPTS="-Dsun.security.krb5.debug=true"), in your log you would
see error like: "unsupported key type found the default TGT: 18"

(which means encryption type 18 per kerberos specification, i.e.
aes256-cts-hmac-sha1-96, isn't supported by Java - due to lack of
unlimited strength JCE)

-Mikhail

On Mon, Mar 16, 2015 at 3:19 PM, Manoj Murumkar
<manoj.murumkar@gmail.com> wrote:
> Thanks Mikhail! It turned out not having unlimited strength JCE files
> everywhere on the cluster was root cause (I reproduced it by leaving
> default files on regionservers on the newly built cluster).
>
> On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar <manoj.murumkar@gmail.com>
> wrote:
>
>> Thanks. We took care of above items (all principals were generated using
>> Ambari tool) although I am not sure about 3rd item. One of my coworkers
>> decided to reinstall HBase, so I am waiting for the cluster to come up to
>> see if we still have the issue. Funny thing is, everything else worked in
>> secure environment (MR, Hive), just not HBase. The error message didn't
>> mention any details why the GSSException was raised (sample below):
>>
>>
>> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: Kerberos principal name is hbase/
>> sfdvgctsn001.xxxxx@SFDVGCT.COM
>>
>> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>>
>> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: Have read input token of size 627 for processing by
>> saslServer.evaluateResponse()
>> *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
>> reading:GSS initiate fail*
>>
>>
>> On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <olorinbant@gmail.com>
>> wrote:
>>
>>> I don't see region server log file in attachment for some reason.. May
>>> be the file is too big and gets rejected by mail server? Without it
>>> it'd be hard to say. Also master log looks a bit incomplete, are there
>>> no more traces? "GSS initiate fail" is normally followed by something
>>> like (just for example)  - "[Caused by GSSException: No valid
>>> credentials provided (Mechanism level: Failed to find any Kerberos
>>> tgt)];" or so. Nothing like that?
>>>
>>> But in general, GSS initialization most often fails for following reasons:
>>>
>>>  - wrong linux path or file permissions on keytab file
>>>  - typo in kerberos principal/realm (that includes lower/upper case
>>> differences)
>>>  - JCE unlimited strength file aren't installed
>>>
>>> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar
>>> <manoj.murumkar@gmail.com> wrote:
>>> > Attached region server log again.
>>> >
>>> > Ted,
>>> >
>>> > Corresponding log for master is:
>>> >
>>> > java.io.IOException: Couldn't setup connection for
>>> > hbase/sfdvgctmn001.gid.gap.com@SFDVGCT.COM to
>>> > hbase/sfdvgctsn001.gid.gap.com@S
>>> >
>>> > FDVGCT.COM
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>>> >
>>> >         at java.security.AccessController.doPrivileged(Native Method)
>>> >
>>> >         at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>>> >
>>> >         at
>>> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
>>> >
>>> >         at
>>> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>>> >
>>> >         at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> >
>>> >         at
>>> >
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> >
>>> >         at java.lang.Thread.run(Thread.java:745)
>>> >
>>> > Caused by:
>>> >
>>> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
>>> > GSS initiate failed
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
>>> >
>>> >         at java.security.AccessController.doPrivileged(Native Method)
>>> >
>>> >         at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> >
>>> >         at
>>> >
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
>>> >
>>> >
>>> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> >>
>>> >> bq. Attached both files
>>> >>
>>> >> I only found hbase-site.xml in the attachment.
>>> >>
>>> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <
>>> manoj.murumkar@gmail.com>
>>> >> wrote:
>>> >>
>>> >> > Attached both files.
>>> >> >
>>> >> >
>>> >> >
>>> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <
>>> olorinbant@gmail.com>
>>> >> > wrote:
>>> >> >
>>> >> >> Hi,
>>> >> >>
>>> >> >> Traces (especially one for region server) look a bit incomplete,
did
>>> >> >> you copy them fully?
>>> >> >>
>>> >> >> Also may help if you post relevant pieces of hbase-site.xml
(with
>>> >> >> security configs).
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Mikhail
>>> >> >>
>>> >> >>
>>> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
>>> >> >> <manoj.murumkar@gmail.com> wrote:
>>> >> >> > Hi,
>>> >> >> >
>>> >> >> > We have a secured cluster. All components are working
well, except
>>> >> >> hbase.
>>> >> >> > Specifically, this is what I see on regionserver:
>>> >> >> >
>>> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: Kerberos principal name is hbase/
>>> >> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
>>> >> >> >
>>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI
>>> >> >> >
>>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: Have read input token of size 627 for processing
by
>>> >> >> > saslServer.evaluateResponse()
>>> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
>>> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception
>>> while
>>> >> >> > reading:GSS initiate fail
>>> >> >> >
>>> >> >> >
>>> >> >> > and on the master:
>>> >> >> >
>>> >> >> >
>>> >> >> > java.io.IOException: Couldn't setup connection for
>>> >> >> > hbase/sfdvgctmn004.
>>> >> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
>>> >> >> >
>>> >> >> >         at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
>>> >> >> >
>>> >> >> >         at java.security.AccessController.doPrivileged(Native
>>> Method)
>>> >> >> >
>>> >> >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
>>> >> >> >
>>> >> >> >         at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>>> >> >> >
>>> >> >> >         at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
>>> >> >> >
>>> >> >> >         at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
>>> >> >> >
>>> >> >> >         at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
>>> >> >> >
>>> >> >> >         at
>>> >> >> >
>>> >> >>
>>> >> >>
>>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
>>> >> >> >
>>> >> >> > All the keytabs are set properly. Has anyone seen this
before?
>>> >> >> Appreciate
>>> >> >> > the time.
>>> >> >> >
>>> >> >> > Manoj
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Thanks,
>>> >> >> Michael Antonov
>>> >> >>
>>> >> >
>>> >> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Michael Antonov
>>>
>>
>>



-- 
Thanks,
Michael Antonov

Mime
View raw message