hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hemanth Yamijala <yhema...@thoughtworks.com>
Subject Re: JobCache directory cleanup
Date Fri, 11 Jan 2013 01:44:07 GMT
Good point. Forgot that one :-)


On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

>
>
> Can you check the job configuration for these ~100 jobs? Do they have
> keep.failed.task.files set to true? If so, these files won't be deleted. If
> it doesn't, it could be a bug.
>
> Sharing your configs for these jobs will definitely help.
>
> Thanks,
> +Vinod
>
>
> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>

Mime
View raw message