spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "StephenZou (JIRA)" <>
Subject [jira] [Updated] (SPARK-22243) streaming job failed to restart from checkpoint
Date Wed, 11 Oct 2017 03:55:00 GMT


StephenZou updated SPARK-22243:
    Component/s:     (was: Spark Core)

> streaming job failed to restart from checkpoint
> -----------------------------------------------
>                 Key: SPARK-22243
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: DStreams
>    Affects Versions: 2.1.0, 2.2.0
>            Reporter: StephenZou
>         Attachments: CheckpointTest.scala
> My spark-defaults.conf has an item related to the issue, I upload all jars in spark's
jars folder to the hdfs path:
> spark.yarn.jars  hdfs:///spark/cache/spark2.2/* 
> Streaming job failed to restart from checkpoint, ApplicationMaster throws  "Error: Could
not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher".  The problem is
always reproducible.
> I examine the sparkconf object recovered from checkpoint, and find spark.yarn.jars are
set empty, which let all jars not exist in AM side. The solution is spark.yarn.jars should
be reload from properties files when recovering from checkpoint. 
> attach is a demo to reproduce the issue.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message