Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 1939D200D27 for ; Wed, 11 Oct 2017 05:42:10 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 17E4D160BE0; Wed, 11 Oct 2017 03:42:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 668F8160BE1 for ; Wed, 11 Oct 2017 05:42:09 +0200 (CEST) Received: (qmail 24857 invoked by uid 500); 11 Oct 2017 03:42:08 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 24836 invoked by uid 99); 11 Oct 2017 03:42:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Oct 2017 03:42:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 9D42D1A17E7 for ; Wed, 11 Oct 2017 03:42:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 9H8ELHbxe8V2 for ; Wed, 11 Oct 2017 03:42:06 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 4B7D65F397 for ; Wed, 11 Oct 2017 03:42:06 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 684DDE06C2 for ; Wed, 11 Oct 2017 03:42:02 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 2D67B25381 for ; Wed, 11 Oct 2017 03:42:01 +0000 (UTC) Date: Wed, 11 Oct 2017 03:42:01 +0000 (UTC) From: "StephenZou (JIRA)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (SPARK-22243) job failed to restart from checkpoint MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 11 Oct 2017 03:42:10 -0000 [ https://issues.apache.org/jira/browse/SPARK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] StephenZou updated SPARK-22243: ------------------------------- Attachment: CheckpointTest.scala the reproducible demo > job failed to restart from checkpoint > ------------------------------------- > > Key: SPARK-22243 > URL: https://issues.apache.org/jira/browse/SPARK-22243 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.1.0, 2.2.0 > Reporter: StephenZou > Attachments: CheckpointTest.scala > > > My spark-defaults.conf has an item related to the issue, I upload all jars in spark's jars folder to the hdfs path: > spark.yarn.jars hdfs:///spark/cache/spark2.2/* > Streaming job failed to restart from checkpoint, ApplicationMaster throws "Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher". The problem is always reproducible. > I examine the sparkconf object recovered from checkpoint, and find spark.yarn.jars are set empty, which let all jars not exist in AM side. The solution is spark.yarn.jars should be reload from properties files when recovering from checkpoint. > attach is a demo to reproduce the issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org