spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewor14 <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-1631] Correctly set the Yarn app name w...
Date Thu, 08 May 2014 04:40:15 GMT
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/539#issuecomment-42512812
  
    I dug through the code a bunch. For those following this PR, here's a quick summary. Looks
like the way we set Spark app name on YARN is actually quite convoluted because there are
two types of app names:
    
    1. The one observed by the RM
    2. The one observed by SparkContext
    
    This PR is mainly concerned with (1) and only on the YARN client mode. Before the changes,
there are two ways of affecting (1), in the order of decreasing priority.
    
    (a) set `spark.app.name` in spark-defaults
    (b) set `SPARK_YARN_APP_NAME` in spark-env
    
    After the changes, this becomes
    
    (a) set `SPARK_YARN_APP_NAME` in spark-env
    (b) set `spark.app.name` in SparkConf (through `conf.setAppName`)
    (c) set `spark.app.name` in spark-defaults
    
    This guarantees that if an environment variable is explicitly specified, then the RM should
use that name. Otherwise, use whatever SparkContext is using for its app name, in which case
(1) == (2). To me, the semantics are pretty clear.
    
    Notice however that in either before or after, there is now way for SparkSubmit's --name
parameter to affect either (1) or (2). This is a separate bug with SparkSubmit that I have
summarized in (SPARK-1755)[https://issues.apache.org/jira/browse/SPARK-1755].


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message