spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <>
Subject [jira] [Commented] (SPARK-7917) Spark doesn't clean up Application Directories (local dirs)
Date Tue, 07 Jul 2015 18:52:04 GMT


Sean Owen commented on SPARK-7917:

Right, this is about standalone. There's but that
was in 1.3. IIRC it looks like this dir gets cleaned up pretty reliably on worker shutdown
if the JVM can exit pretty normally, so I think the question is, does it still happen on master?
and wha causes the normal code path not to happen?

> Spark doesn't clean up Application Directories (local dirs) 
> ------------------------------------------------------------
>                 Key: SPARK-7917
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.3.0
>            Reporter: Zach Fry
>            Priority: Minor
> Similar to SPARK-4834. 
> Spark does clean up the cache and lock files in the local dirs, however, it doesn't clean
up the actual directories. 
> We have to write custom scripts to go back through the local dirs and find directories
that don't contain any files and clear those out. 
> Its a pretty simple repro: 
> Run a job that does some shuffling, wait for the shuffle files to get cleaned up, go
and look on disk at spark.local.dir and notice that the directory(s) are still there, but
there are no files in them.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message