hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdelrhman Shettia <ashet...@hortonworks.com>
Subject Re: Auto clean DistCache?
Date Tue, 26 Mar 2013 19:53:49 GMT
Hi JM ,

Actually these dirs need to be purged by a script that keeps the last 2 days worth of files,
Otherwise you may run into # of open files exceeds error. 

Thanks


On Mar 25, 2013, at 5:16 PM, Jean-Marc Spaggiari <jean-marc@spaggiari.org> wrote:

> Hi,
> 
> Each time my MR job is run, a directory is created on the TaskTracker
> under mapred/local/taskTracker/hadoop/distcache (based on my
> configuration).
> 
> I looked at the directory today, and it's hosting thousands of
> directories and more than 8GB of data there.
> 
> Is there a way to automatically delete this directory when the job is done?
> 
> Thanks,
> 
> JM


Mime
View raw message