hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nkechi Achara <nkach...@googlemail.com>
Subject Re: hbase/spark - Delegation Token can be issued only with kerberos or web authentication
Date Fri, 18 Nov 2016 14:00:57 GMT
Can you use the principal and keytab options in Spark submit? These should
circumvent this issue.

On 18 Nov 2016 1:01 p.m., "Abel Fernández" <mevsmyself@gmail.com> wrote:

> Hello,
>
> We are having problems with the delegation of the token in a secure
> cluster: Delegation Token can be issued only with kerberos or web
> authentication
>
> We have a spark process which is generating the hfiles to be loaded into
> hbase. To generate these hfiles, (we are using a back-ported version of the
> latest hbase/spark code), we are using this method HBaseRDDFunctions.
> hbaseBulkLoadThinRows.
>
> I think the problem is in the below piece of code. This function is
> executed in every partition of the rdd, when the executors are trying to
> execute the code, the executors do not have a valid kerberos credential and
> cannot execute anything.
>
> private def hbaseForeachPartition[T](configBroadcast:
>                                        Broadcast[SerializableWritable[
> Configuration]],
>                                         it: Iterator[T],
>                                         f: (Iterator[T], Connection) =>
> Unit) = {
>
>     val config = getConf(configBroadcast)
>
>     applyCreds
>     // specify that this is a proxy user
>     val smartConn = HBaseConnectionCache.getConnection(config)
>     f(it, smartConn.connection)
>     smartConn.close()
>   }
>
> I have attached the spark-submit and the complete error log trace. Has
> anyone faced this problem before?
>
> Thanks in advance.
>
> Regards,
> Abel.
> --
> Un saludo - Best Regards.
> Abel
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message