spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wilfred Spiegelenburg (JIRA)" <>
Subject [jira] [Created] (SPARK-7705) Cleanup of .sparkStaging directory fails if application is killed
Date Mon, 18 May 2015 12:33:00 GMT
Wilfred Spiegelenburg created SPARK-7705:

             Summary: Cleanup of .sparkStaging directory fails if application is killed
                 Key: SPARK-7705
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 1.3.0
            Reporter: Wilfred Spiegelenburg

When a streaming application is killed while running on YARN the .sparkStaging directory is
not cleaned up. Setting spark.yarn.preserve.staging.files=false does not help and still leaves
the files around.

The changes in SPARK-7503 do not catch this case since there is no exception in the shutdown.
When the application gets killed the AM gets told to shutdown and the shutdown hook is run
but the clean up is not triggered.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message