hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Murumkar <manoj.murum...@gmail.com>
Subject Re: Kerberos exception
Date Mon, 16 Mar 2015 23:25:03 GMT
I did that, to no avail. I am surprised by level of verbosity (lack
thereof) too, something new you learn every day :)
On Mar 16, 2015 3:44 PM, "Mikhail Antonov" <olorinbant@gmail.com> wrote:

> Glad it helped!
>
> In general though, in this cases, if you turn on java kerberos debug
> logging (via something like export
> HADOOP_OPTS="-Dsun.security.krb5.debug=true"), in your log you would
> see error like: "unsupported key type found the default TGT: 18"
>
> (which means encryption type 18 per kerberos specification, i.e.
> aes256-cts-hmac-sha1-96, isn't supported by Java - due to lack of
> unlimited strength JCE)
>
> -Mikhail
>
> On Mon, Mar 16, 2015 at 3:19 PM, Manoj Murumkar
> <manoj.murumkar@gmail.com> wrote:
> > Thanks Mikhail! It turned out not having unlimited strength JCE files
> > everywhere on the cluster was root cause (I reproduced it by leaving
> > default files on regionservers on the newly built cluster).
> >
> > On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar <
> manoj.murumkar@gmail.com>
> > wrote:
> >
> >> Thanks. We took care of above items (all principals were generated using
> >> Ambari tool) although I am not sure about 3rd item. One of my coworkers
> >> decided to reinstall HBase, so I am waiting for the cluster to come up
> to
> >> see if we still have the issue. Funny thing is, everything else worked
> in
> >> secure environment (MR, Hive), just not HBase. The error message didn't
> >> mention any details why the GSSException was raised (sample below):
> >>
> >>
> >> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: Kerberos principal name is hbase/
> >> sfdvgctsn001.xxxxx@SFDVGCT.COM
> >>
> >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: Created SASL server with mechanism = GSSAPI
> >>
> >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: Have read input token of size 627 for processing by
> >> saslServer.evaluateResponse()
> >> *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while
> >> reading:GSS initiate fail*
> >>
> >>
> >> On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <olorinbant@gmail.com>
> >> wrote:
> >>
> >>> I don't see region server log file in attachment for some reason.. May
> >>> be the file is too big and gets rejected by mail server? Without it
> >>> it'd be hard to say. Also master log looks a bit incomplete, are there
> >>> no more traces? "GSS initiate fail" is normally followed by something
> >>> like (just for example)  - "[Caused by GSSException: No valid
> >>> credentials provided (Mechanism level: Failed to find any Kerberos
> >>> tgt)];" or so. Nothing like that?
> >>>
> >>> But in general, GSS initialization most often fails for following
> reasons:
> >>>
> >>>  - wrong linux path or file permissions on keytab file
> >>>  - typo in kerberos principal/realm (that includes lower/upper case
> >>> differences)
> >>>  - JCE unlimited strength file aren't installed
> >>>
> >>> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar
> >>> <manoj.murumkar@gmail.com> wrote:
> >>> > Attached region server log again.
> >>> >
> >>> > Ted,
> >>> >
> >>> > Corresponding log for master is:
> >>> >
> >>> > java.io.IOException: Couldn't setup connection for
> >>> > hbase/sfdvgctmn001.gid.gap.com@SFDVGCT.COM to
> >>> > hbase/sfdvgctsn001.gid.gap.com@S
> >>> >
> >>> > FDVGCT.COM
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >>> >
> >>> >         at java.security.AccessController.doPrivileged(Native Method)
> >>> >
> >>> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >>> >
> >>> >         at
> >>> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743)
> >>> >
> >>> >         at
> >>> >
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
> >>> >
> >>> >         at
> >>> >
> >>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>> >
> >>> >         at
> >>> >
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>> >
> >>> >         at java.lang.Thread.run(Thread.java:745)
> >>> >
> >>> > Caused by:
> >>> >
> >>>
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> >>> > GSS initiate failed
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
> >>> >
> >>> >         at java.security.AccessController.doPrivileged(Native Method)
> >>> >
> >>> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> >
> >>> >         at
> >>> >
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
> >>> >
> >>> >
> >>> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <yuzhihong@gmail.com>
wrote:
> >>> >>
> >>> >> bq. Attached both files
> >>> >>
> >>> >> I only found hbase-site.xml in the attachment.
> >>> >>
> >>> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar <
> >>> manoj.murumkar@gmail.com>
> >>> >> wrote:
> >>> >>
> >>> >> > Attached both files.
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov <
> >>> olorinbant@gmail.com>
> >>> >> > wrote:
> >>> >> >
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >> Traces (especially one for region server) look a bit incomplete,
> did
> >>> >> >> you copy them fully?
> >>> >> >>
> >>> >> >> Also may help if you post relevant pieces of hbase-site.xml
(with
> >>> >> >> security configs).
> >>> >> >>
> >>> >> >> Thanks,
> >>> >> >> Mikhail
> >>> >> >>
> >>> >> >>
> >>> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar
> >>> >> >> <manoj.murumkar@gmail.com> wrote:
> >>> >> >> > Hi,
> >>> >> >> >
> >>> >> >> > We have a secured cluster. All components are working
well,
> except
> >>> >> >> hbase.
> >>> >> >> > Specifically, this is what I see on regionserver:
> >>> >> >> >
> >>> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: Kerberos principal name is hbase/
> >>> >> >> > sfdvgctsn001.xxxxx@SFDVGCT.COM
> >>> >> >> >
> >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: Created SASL server with mechanism
= GSSAPI
> >>> >> >> >
> >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: Have read input token of size 627
for
> processing by
> >>> >> >> > saslServer.evaluateResponse()
> >>> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020]
> >>> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught
exception
> >>> while
> >>> >> >> > reading:GSS initiate fail
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > and on the master:
> >>> >> >> >
> >>> >> >> >
> >>> >> >> > java.io.IOException: Couldn't setup connection for
> >>> >> >> > hbase/sfdvgctmn004.
> >>> >> >> > xxxxx@SFDVGCT.COM to hbase/sfdvgctsn001.xxxxx@SFDVGCT.COM
> >>> >> >> >
> >>> >> >> >         at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869)
> >>> >> >> >
> >>> >> >> >         at java.security.AccessController.doPrivileged(Native
> >>> Method)
> >>> >> >> >
> >>> >> >> >         at javax.security.auth.Subject.doAs(Subject.java:415)
> >>> >> >> >
> >>> >> >> >         at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> >>> >> >> >
> >>> >> >> >         at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841)
> >>> >> >> >
> >>> >> >> >         at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951)
> >>> >> >> >
> >>> >> >> >         at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
> >>> >> >> >
> >>> >> >> >         at
> >>> >> >> >
> >>> >> >>
> >>> >> >>
> >>>
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
> >>> >> >> >
> >>> >> >> > All the keytabs are set properly. Has anyone seen
this before?
> >>> >> >> Appreciate
> >>> >> >> > the time.
> >>> >> >> >
> >>> >> >> > Manoj
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> --
> >>> >> >> Thanks,
> >>> >> >> Michael Antonov
> >>> >> >>
> >>> >> >
> >>> >> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Thanks,
> >>> Michael Antonov
> >>>
> >>
> >>
>
>
>
> --
> Thanks,
> Michael Antonov
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message