hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhong,Zhenyu" <edwardzh...@baidu.com>
Subject yarn spark job submit problem
Date Thu, 27 Oct 2016 22:58:20 GMT
Hi,

I am using spark-1.5.1 and hadoop-2.6.4. After completing the configuration, I am able to
run spark job via YARN.
However, I have a problem that YARN often could’t find the local zip-archive generated during
the preparation stage.
I submitted the job using the client mode. Thus the zip-archive was prepared on the machine
which runs the client process.

According to the following log, the zip-archive was there on the local filesystem, however,
yarn couldn’t find it and report “File Not Found”.

Has anyone have similar problems? Any solution for that?

Thanks in advance.

edward


16/10/27 15:17:46 INFO yarn.Client: Will allocate AM container, with 896 MB memory including
384 MB overhead

16/10/27 15:17:46 INFO yarn.Client: Setting up container launch context for our AM

16/10/27 15:17:46 INFO yarn.Client: Setting up the launch environment for our AM container

16/10/27 15:17:46 INFO yarn.Client: Preparing resources for our AM container

16/10/27 15:17:46 INFO yarn.Client: Source and destination file systems are the same. Not
copying file:/home/spark/spark-1.5.1-bin-hadoop2.6/lib/spark-assembly-1.5.1-hadoop2.6.0.jar

16/10/27 15:17:46 INFO yarn.Client: Source and destination file systems are the same. Not
copying file:/home/spark/spark-1.5.1-bin-hadoop2.6/python/lib/pyspark.zip

16/10/27 15:17:46 INFO yarn.Client: Source and destination file systems are the same. Not
copying file:/home/spark/spark-1.5.1-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip

16/10/27 15:17:46 INFO yarn.Client: Source and destination file systems are the same. Not
copying file:/home/spark/spark-1.5.1-bin-hadoop2.6/local/spark-b9c155cf-4624-4c68-9d0d-d3b9d5748601/__spark_conf__6786864409197988390.zip

16/10/27 15:17:46 INFO spark.SecurityManager: Changing view acls to: spark

16/10/27 15:17:46 INFO spark.SecurityManager: Changing modify acls to: spark

16/10/27 15:17:46 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui
acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)

16/10/27 15:17:46 INFO yarn.Client: Submitting application 2371 to ResourceManager

16/10/27 15:17:46 INFO impl.YarnClientImpl: Submitted application application_1474479656852_2371

16/10/27 15:17:47 INFO yarn.Client: Application report for application_1474479656852_2371
(state: ACCEPTED)

16/10/27 15:17:47 INFO yarn.Client:

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: default

start time: 1477606690545

final status: UNDEFINED

tracking URL: http://xlab-node4.usdc.baidu.com:8088/proxy/application_1474479656852_2371/

user: spark

16/10/27 15:17:48 INFO yarn.Client: Application report for application_1474479656852_2371
(state: ACCEPTED)

16/10/27 15:17:49 INFO yarn.Client: Application report for application_1474479656852_2371
(state: FAILED)

16/10/27 15:17:49 INFO yarn.Client:

client token: N/A

diagnostics: Application application_1474479656852_2371 failed 2 times due to AM Container
for appattempt_1474479656852_2371_000002 exited with  exitCode: -1000

For more detailed output, check application tracking page:http://xlab-node4.usdc.baidu.com:8088/proxy/application_1474479656852_2371/Then,
click on links to logs of each attempt.

Diagnostics: File file:/home/spark/spark-1.5.1-bin-hadoop2.6/local/spark-b9c155cf-4624-4c68-9d0d-d3b9d5748601/__spark_conf__6786864409197988390.zip
does not exist

java.io.FileNotFoundException: File file:/home/spark/spark-1.5.1-bin-hadoop2.6/local/spark-b9c155cf-4624-4c68-9d0d-d3b9d5748601/__spark_conf__6786864409197988390.zip
does not exist


Mime
View raw message