spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "StephenZou (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-22242) job failed to restart from checkpoint
Date Wed, 11 Oct 2017 03:39:00 GMT
StephenZou created SPARK-22242:
----------------------------------

             Summary: job failed to restart from checkpoint
                 Key: SPARK-22242
                 URL: https://issues.apache.org/jira/browse/SPARK-22242
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.2.0, 2.1.0
            Reporter: StephenZou


My spark-defaults.conf has an item related to the issue, I upload all jars in spark's jars
folder to the hdfs path:
spark.yarn.jars  hdfs:///spark/cache/spark2.2/* 

Streaming job failed to restart from checkpoint, ApplicationMaster throws  "Error: Could not
find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher".  The problem is always
reproducible.

I examine the sparkconf object recovered from checkpoint, and find spark.yarn.jars are set
empty, which let all jars not exist in AM side. The solution is spark.yarn.jars should be
reload from properties files when recovering from checkpoint. 

attach is a demo to reproduce the issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message