spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saisai Shao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19992) spark-submit on deployment-mode cluster
Date Fri, 24 Mar 2017 05:56:41 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939867#comment-15939867
] 

Saisai Shao commented on SPARK-19992:
-------------------------------------

Sorry I cannot give you valid suggestions without knowing your actual environment. Basically
running Spark on yarn you don't have to configure anything except HADOOP_CONF_DIR specified
in spark-env.sh. Other than this default configurations should be enough.

You could also send your problem to mail list, I think there will be more users who met the
same problem before. Here JIRA is mainly used to track dev work of Spark, not for questions.

> spark-submit on deployment-mode cluster
> ---------------------------------------
>
>                 Key: SPARK-19992
>                 URL: https://issues.apache.org/jira/browse/SPARK-19992
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.0.2
>         Environment: spark version 2.0.2
> hadoop version 2.6.0
>            Reporter: narendra maru
>
> spark version 2.0.2
> hadoop version 2.6.0
> spark -submit command
> "spark-submit --class spark.mongohadoop.testing3 --master yarn --deploy-mode cluster
--jars /home/ec2-user/jars/hgmongonew.jar, /home/ec2-user/jars/mongo-hadoop-spark-2.0.1.jar"
> after adding following in
> 1 Spark-default.conf
> spark.executor.extraJavaOptions -Dconfig.fuction.conf 
> spark.yarn.jars=local:/usr/local/spark-2.0.2-bin-hadoop2.6/yarn/*
> spark.eventLog.dir=hdfs://localhost:9000/user/spark/applicationHistory
> spark.eventLog.enabled=true
> 2yarn-site.xml
> <property>
> <name>yarn.application.classpath</name>
> <value>
> /usr/local/hadoop-2.6.0/etc/hadoop,
> /usr/local/hadoop-2.6.0/,
> /usr/local/hadoop-2.6.0/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/common/,
> /usr/local/hadoop-2.6.0/share/hadoop/common/lib/
> /usr/local/hadoop-2.6.0/share/hadoop/hdfs/,
> /usr/local/hadoop-2.6.0/share/hadoop/hdfs/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/,
> /usr/local/hadoop-2.6.0/share/hadoop/mapreduce/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/tools/lib/,
> /usr/local/hadoop-2.6.0/share/hadoop/yarn/,
> /usr/local/hadoop-2.6.0/share/hadoop/yarn/lib/*,
> /usr/local/spark-2.0.2-bin-hadoop2.6/jars/spark-yarn_2.11-2.0.2.jar
> </value>
> </property>
> Error on log:-
> Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
> Error on terminal:-
> diagnostics: Application application_1489673977198_0002 failed 2 times due to AM Container
for appattempt_1489673977198_0002_000002 exited with exitCode: 1 For more detailed output,
check application tracking page:http://bdg-hdp-sparkmaster:8088/proxy/application_1489673977198_0002/Then,
click on links to logs of each attempt.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message