spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <ja...@japila.pl>
Subject Re: submitting spark job with kerberized Hadoop issue
Date Sat, 06 Aug 2016 10:00:46 GMT
Hi Aneela,

I don't really know. I've never been using (or even toying with) Spark
Standalone to access a secured HDFS cluster. I however think the
settings won't work since they are for Spark on YARN (I would not be
surprised to know that it is not supported outside Spark on YARN).

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sat, Aug 6, 2016 at 11:03 AM, Aneela Saleem <aneela@platalytics.com> wrote:
> Hi Jacek!
>
> Thanks for your response. I am using spark standalone. I have secured hadoop
> cluster, Can you please guide me  wha to do if i want to access hadoop in my
> spark job?
>
> Thanks
>
> On Sat, Aug 6, 2016 at 12:34 AM, Jacek Laskowski <jacek@japila.pl> wrote:
>>
>> Just to make things clear...are you using Spark Standalone and Spark
>> on YARN-specific settings? I don't think it's gonna work.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> ----
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Fri, Aug 5, 2016 at 3:54 PM, Aneela Saleem <aneela@platalytics.com>
>> wrote:
>> > Hi all,
>> >
>> > I'm trying to connect to Kerberized Hadoop cluster using spark job. I
>> > have
>> > kinit'd from command line. When i run the following job i.e.,
>> >
>> > ./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
>> > spark/hadoop-master@platalyticsrealm --class
>> > com.platalytics.example.spark.App --master spark://hadoop-master:7077
>> > /home/vm6/project-1-jar-with-dependencies.jar
>> > hdfs://hadoop-master:8020/text
>> >
>> > I get the error:
>> >
>> > Caused by: java.io.IOException:
>> > org.apache.hadoop.security.AccessControlException: Client cannot
>> > authenticate via:[TOKEN, KERBEROS]
>> > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >
>> > Following are the contents of spark-defaults.conf file:
>> >
>> > spark.master                     spark://hadoop-master:7077
>> > spark.eventLog.enabled           true
>> > spark.eventLog.dir               hdfs://hadoop-master:8020/spark/logs
>> > spark.serializer
>> > org.apache.spark.serializer.KryoSerializer
>> > spark.yarn.access.namenodes hdfs://hadoop-master:8020/
>> > spark.yarn.security.tokens.hbase.enabled true
>> > spark.yarn.security.tokens.hive.enabled true
>> > spark.yarn.principal yarn/hadoop-master@platalyticsrealm
>> > spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
>> >
>> >
>> > Also i have added following in spark-env.sh file:
>> >
>> > HOSTNAME=`hostname -f`
>> > export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
>> > -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
>> > -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
>> >
>> >
>> > Please guide me, how to trace the issue?
>> >
>> > Thanks
>> >
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message