Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 24 Oct 2016 07:29:58 +0000 (UTC)
From: "David Shar (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13013822.1476961523000.64583.1477294198488@Atlassian.JIRA>
In-Reply-To: <JIRA.13013822.1476961523000@Atlassian.JIRA>
References: <JIRA.13013822.1476961523000@Atlassian.JIRA> <JIRA.13013822.1476961523729@arcas>
Subject: [jira] [Commented] (SPARK-18027) .sparkStaging not clean on RM
 ApplicationNotFoundException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 24 Oct 2016 07:30:00 -0000


    [ https://issues.apache.org/jira/browse/SPARK-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601231#comment-15601231 ] 

David Shar commented on SPARK-18027:
------------------------------------

Yes, I believe it is safer, we cannot be sure what Yarn is doing on connection failure.


> .sparkStaging not clean on RM ApplicationNotFoundException
> ----------------------------------------------------------
>
>                 Key: SPARK-18027
>                 URL: https://issues.apache.org/jira/browse/SPARK-18027
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.6.0
>            Reporter: David Shar
>            Priority: Minor
>
> Hi,
> It seems that SPARK-7705 didn't fix all issues with .sparkStaging folder cleanup.
> in Client.scala:monitorApplication 
> {code}
>  val report: ApplicationReport =
>         try {
>           getApplicationReport(appId)
>         } catch {
>           case e: ApplicationNotFoundException =>
>             logError(s"Application $appId not found.")
>             return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
>           case NonFatal(e) =>
>             logError(s"Failed to contact YARN for application $appId.", e)
>             return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
>         }
> ....
> if (state == YarnApplicationState.FINISHED ||
>         state == YarnApplicationState.FAILED ||
>         state == YarnApplicationState.KILLED) {
>         cleanupStagingDir(appId)
>         return (state, report.getFinalApplicationStatus)
>  }
> {code}
> In case of ApplicationNotFoundException, we don't cleanup the sparkStaging folder.
> I believe we should call cleanupStagingDir(appId) on the catch clause above.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org