spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Shar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-18027) .sparkStaging not clean on RM ApplicationNotFoundException
Date Mon, 24 Oct 2016 07:29:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601231#comment-15601231
] 

David Shar commented on SPARK-18027:
------------------------------------

Yes, I believe it is safer, we cannot be sure what Yarn is doing on connection failure.


> .sparkStaging not clean on RM ApplicationNotFoundException
> ----------------------------------------------------------
>
>                 Key: SPARK-18027
>                 URL: https://issues.apache.org/jira/browse/SPARK-18027
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.6.0
>            Reporter: David Shar
>            Priority: Minor
>
> Hi,
> It seems that SPARK-7705 didn't fix all issues with .sparkStaging folder cleanup.
> in Client.scala:monitorApplication 
> {code}
>  val report: ApplicationReport =
>         try {
>           getApplicationReport(appId)
>         } catch {
>           case e: ApplicationNotFoundException =>
>             logError(s"Application $appId not found.")
>             return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED)
>           case NonFatal(e) =>
>             logError(s"Failed to contact YARN for application $appId.", e)
>             return (YarnApplicationState.FAILED, FinalApplicationStatus.FAILED)
>         }
> ....
> if (state == YarnApplicationState.FINISHED ||
>         state == YarnApplicationState.FAILED ||
>         state == YarnApplicationState.KILLED) {
>         cleanupStagingDir(appId)
>         return (state, report.getFinalApplicationStatus)
>  }
> {code}
> In case of ApplicationNotFoundException, we don't cleanup the sparkStaging folder.
> I believe we should call cleanupStagingDir(appId) on the catch clause above.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message