spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-12800) Subtle bug on Spark Yarn Client under Kerberos Security Mode
Date Tue, 29 Mar 2016 16:09:25 GMT

    [ https://issues.apache.org/jira/browse/SPARK-12800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216243#comment-15216243
] 

Thomas Graves commented on SPARK-12800:
---------------------------------------

You are talking about launching a job using org.apache.spark.deploy.yarn.Client directly,
correct?  If so, we don't officially support that, I realize it isn't currently private and
some people are using it but in 2.0 that will be made private.



> Subtle bug on Spark Yarn Client under Kerberos Security Mode
> ------------------------------------------------------------
>
>                 Key: SPARK-12800
>                 URL: https://issues.apache.org/jira/browse/SPARK-12800
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.5.1, 1.5.2
>            Reporter: Chester
>
> Version used: Spark 1.5.1 (1.5.2-SNAPSHOT) 
> Deployment Mode: Yarn-Cluster
> Problem observed: 
>   When running spark job directly from YarnClient (without using spark-submit, I did
not verify the spark-submit has the same issue or not), when kerberos security is enabled,
the first time run spark job always fail. The failure is due to that the hadoop consider the
job is in SIMPLE model rather than Kerberos mode.  But without shutting down the JVM, run
the same job again, the spark job will pass. If one restart the JVM, then the spark job will
fail again. 
> The cause: 
>   Tracking down the source of the issue, I found that the problem seems lie at the spark
Yarn Client.scala. In the Client
> {code}
> def prepareLocalResources() method  L 266 of Client.java, the following line code is
called. 
>  YarnSparkHadoopUtil.get.obtainTokensForNamenodes(nns, hadoopConf, credentials)
> {code}
> The YarnSparkHadoopUtil.get is in turns get initialized via reflection
> {code}
> object SparkHadoopUtil {
>   private val hadoop = {
>     val yarnMode = java.lang.Boolean.valueOf(
>         System.getProperty("SPARK_YARN_MODE", System.getenv("SPARK_YARN_MODE")))
>     if (yarnMode) {
>       try {
>         Utils.classForName("org.apache.spark.deploy.yarn.YarnSparkHadoopUtil")
>           .newInstance()
>           .asInstanceOf[SparkHadoopUtil]
>       } catch {
>        case e: Exception => throw new SparkException("Unable to load YARN support",
e)
>       }
>     } else {
>       new SparkHadoopUtil
>     }
>   } 
>   def get: SparkHadoopUtil = {
>     hadoop
>   }
> }
>  
> class SparkHadoopUtil extends Logging {
>   private val sparkConf = new SparkConf()
>   val conf: Configuration = newConfiguration(sparkConf)
>   UserGroupInformation.setConfiguration(conf)
>    .... rest of line
> }
> {code}
> Here SparkHadoopUtil creates a empty SparkConf and Hadoop Configuration from that and
set to UserGroupInformation
> {code}
>   UserGroupInformation.setConfiguration(conf)
> {code}
>   As the UserGroupInformation.authenticationMethod is static, above all wipe out the
security settings. UserGroupInformation.isSecurityEnabled() changed from true to false. Thus
the sequence call will fail. 
>  Since the SparkHadoopUtil.hadoop is static/non-mutable variable, so 
> the next run it will be not create again, then UserGroupInformation.setConfiguration(conf)

> will not be called again, so the sequence spark job works. 
> The work around: 
> {code}
>         //first initialize the SparkHadoopUtil, which will create a static instance
>         //which will set UserGroupInformation to a empty hadoop Configuration.
>         //we will need to reset the UserGroupInformation after that.
>         val util = SparkHadoopUtil.get
>         UserGroupInformation.setConfiguration(hadoopConf)
> {code}
>       Then call 
>           client.run() 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message