hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abel Fernández <mevsmys...@gmail.com>
Subject Re: hbase/spark - Delegation Token can be issued only with kerberos or web authentication
Date Fri, 18 Nov 2016 14:10:00 GMT
Hi Nkechi,

Thank for your early response.

I am currently specifying the principal and the keytab in the spark-submit,
the keytab is in the same location in every node manager.

SPARK_CONF_DIR=conf-hbase spark-submit --master yarn-cluster \
  --executor-memory 6G \
  --num-executors 10 \
  --queue cards \
  --executor-cores 4 \
  --driver-java-options "-Dlog4j.configuration=file:log4j.properties" \
  --driver-class-path "$2" \
  --jars file:/opt/orange/lib/rocksdbjni-4.5.1.jar \
  --conf
"spark.driver.extraClassPath=/var/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.2.0-incubating.jar:/var/cloudera/parcels/CDH/jars/hbase-server-1.0.0-cdh5.5.4.jar:/var/cloudera/parcels/CDH/jars/hbase-common-1.0.0-cdh5.5.4.jar:/var/cloudera/parcels/CDH/lib/hbase/lib/hbase-client-1.0.0-cdh5.5.4.jar:/var/cloudera/parcels/CDH/lib/hbase/lib/hbase-protocol-1.0.0-cdh5.5.4.jar:/opt/orange/lib/rocksdbjni-4.5.1.jar:/var/cloudera/parcels/CLABS_PHOENIX-4.5.2-1.clabs_phoenix1.2.0.p0.774/lib/phoenix/lib/phoenix-core-1.2.0.jar:/var/cloudera/parcels/CDH/jars/hadoop-mapreduce-client-core-2.6.0-cdh5.5.4.jar"
\
  --conf
"spark.executor.extraClassPath=/var/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.2.0-incubating.jar:/var/cloudera/parcels/CDH/jars/hbase-server-1.0.0-cdh5.5.4.jar:/var/cloudera/parcels/CDH/jars/hbase-common-1.0.0-cdh5.5.4.jar:/var/cloudera/parcels/CDH/lib/hbase/lib/hbase-client-1.0.0-cdh5.5.4.jar:/var/cloudera/parcels/CDH/lib/hbase/lib/hbase-protocol-1.0.0-cdh5.5.4.jar:/opt/orange/lib/rocksdbjni-4.5.1.jar:/var/cloudera/parcels/CLABS_PHOENIX-4.5.2-1.clabs_phoenix1.2.0.p0.774/lib/phoenix/lib/phoenix-core-1.2.0.jar:/var/cloudera/parcels/CDH/jars/hadoop-mapreduce-client-core-2.6.0-cdh5.5.4.jar"\
  --principal hbase@COMPANY.CORP \
  --keytab /opt/company/conf/hbase.keytab \
  --files
"owl.properties,conf-hbase/log4j.properties,conf-hbase/hbase-site.xml,conf-hbase/core-site.xml,$2"
\
  --class $1 \
  cards-batch-$3-jar-with-dependencies.jar $2

On Fri, 18 Nov 2016 at 14:01 Nkechi Achara <nkachara@googlemail.com> wrote:

> Can you use the principal and keytab options in Spark submit? These should
> circumvent this issue.
>
> On 18 Nov 2016 1:01 p.m., "Abel Fernández" <mevsmyself@gmail.com> wrote:
>
> > Hello,
> >
> > We are having problems with the delegation of the token in a secure
> > cluster: Delegation Token can be issued only with kerberos or web
> > authentication
> >
> > We have a spark process which is generating the hfiles to be loaded into
> > hbase. To generate these hfiles, (we are using a back-ported version of
> the
> > latest hbase/spark code), we are using this method HBaseRDDFunctions.
> > hbaseBulkLoadThinRows.
> >
> > I think the problem is in the below piece of code. This function is
> > executed in every partition of the rdd, when the executors are trying to
> > execute the code, the executors do not have a valid kerberos credential
> and
> > cannot execute anything.
> >
> > private def hbaseForeachPartition[T](configBroadcast:
> >                                        Broadcast[SerializableWritable[
> > Configuration]],
> >                                         it: Iterator[T],
> >                                         f: (Iterator[T], Connection) =>
> > Unit) = {
> >
> >     val config = getConf(configBroadcast)
> >
> >     applyCreds
> >     // specify that this is a proxy user
> >     val smartConn = HBaseConnectionCache.getConnection(config)
> >     f(it, smartConn.connection)
> >     smartConn.close()
> >   }
> >
> > I have attached the spark-submit and the complete error log trace. Has
> > anyone faced this problem before?
> >
> > Thanks in advance.
> >
> > Regards,
> > Abel.
> > --
> > Un saludo - Best Regards.
> > Abel
> >
>
-- 
Un saludo - Best Regards.
Abel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message