hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From REYANE OUKPEDJO <r.oukpe...@yahoo.com.INVALID>
Subject Issuse to get oozi/spark to read from a local cluster and write back to a remote cluster
Date Thu, 04 Aug 2016 14:45:58 GMT
Hi there,
I am having issues to get oozie spark action to read from hdfs  on one secured cluster and
write it back to an hdfs in a remote secured  cluster with kerberos . However ooze/spark
can read from the hdfs in a local cluster and write back to hdfs in the same cluster . the
oozie job properties file has the following :
sparkOpts=--conf spark.yarn.am.extraJavaOptions=-Diop.version={{iop_full_version}} --conf
spark.yarn.access.namenodes=hdfs://namenode1:8020,hdfs://namenode2:8020 

It seems that oozie is not attempting  to get the delegation token for the second namenode
which a remote namenode although the abov option is set. Is it any other way to force oozie
to request the HDFS delegation token for both name nodes? please help.


Also notice a regular spark can be able to perform the above




Mime
View raw message