hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: JobCache directory cleanup
Date Thu, 10 Jan 2013 17:23:28 GMT
Can you check the job configuration for these ~100 jobs? Do they have
keep.failed.task.files set to true? If so, these files won't be deleted. If
it doesn't, it could be a bug.

Sharing your configs for these jobs will definitely help.

Thanks,
+Vinod


On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov
<itretyakov@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>

Mime
View raw message