spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Resolved] (SPARK-7439) Should delete temporary local directories
Date Wed, 20 May 2015 08:17:59 GMT


Sean Owen resolved SPARK-7439.
    Resolution: Duplicate

Not sure what to do with this one as the dirs should already be cleaned up on JVM exit, certainly
as of 1.4, and certainly if the context stops cleanly. This doesn't mean there aren't failure
cases where these might be left, since nothing can always clean up in case of a hard failure.
Issues like SPARK-7503 which are tackling specific common cases of this might be the best
way forward.

> Should delete temporary local directories
> -----------------------------------------
>                 Key: SPARK-7439
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager
>    Affects Versions: 1.3.1
>         Environment: Windows 7, CentOS 6.6
>            Reporter: Taeyun Kim
>            Priority: Minor
> Spark does not delete temporary local directories.
> After a spark program completes, there are 3 temporary directories remain in the temp
directory. The directory names are like this: spark-2e389487-40cc-4a82-a5c7-353c0feefbb7
> The directories are empty.
> They are created every time the Spark program runs. So the number of files and directories
keeps growing.
> I've traced the spark source code.
> The module methods that create the 3 'temp' directories are as follows:
> * DiskBlockManager.createLocalDirs
> * HttpFileServer.initialize
> * SparkEnv.sparkFilesDir
> They (eventually) call Utils.getOrCreateLocalRootDirs and then Utils.createDirectory,
which intentionally does NOT mark the directory for automatic deletion.
> The comment of createDirectory method says: "The directory is guaranteed to be newly
created, and is not marked for automatic deletion."
> But since the directories does not hold useful data after the program completes, they
should be deleted if possible.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message