Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F41C6E29B for ; Fri, 11 Jan 2013 10:21:34 +0000 (UTC) Received: (qmail 67065 invoked by uid 500); 11 Jan 2013 10:21:30 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 66742 invoked by uid 500); 11 Jan 2013 10:21:29 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 66701 invoked by uid 99); 11 Jan 2013 10:21:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jan 2013 10:21:28 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hemanty@thoughtworks.com designates 64.18.0.184 as permitted sender) Received: from [64.18.0.184] (HELO exprod5og107.obsmtp.com) (64.18.0.184) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jan 2013 10:21:21 +0000 Received: from mail-ie0-f197.google.com ([209.85.223.197]) (using TLSv1) by exprod5ob107.postini.com ([64.18.4.12]) with SMTP ID DSNKUO/ni0qYS9TndkNSLXrgt+21JW97agKB@postini.com; Fri, 11 Jan 2013 02:21:00 PST Received: by mail-ie0-f197.google.com with SMTP id 16so7521337iea.4 for ; Fri, 11 Jan 2013 02:20:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:date:message-id :subject:from:to:content-type:x-gm-message-state; bh=vup6gSuS5m4RDbQYWxf3OF7e6wPtXonXJrqMvFlAHDY=; b=eQ4uBiip9nCPD0vKN3ZNdfWZipFcyFXTUXxk5kZdQDBPyVl1b8s/11pnugC4LWFErX r1SFnTP70Yw06TEuPQeEq+BT8cLcMxMaemOrOlqN9tkzILsPnZcAc4yDlIvlZ/ovlqn5 WoXrcrn5qsFZmiYU9WJhiXGXzHfDIhbYJU+DjisWhAiP4ukuzydFlIgbM5ShNrUBo5Sx NTbFAtofpdUPAtmb96p3LJcOKf3G5zqj/lNtZKiv30paklXxRw/z1tKU8AHuRorfutQM NP1qFFUNqDvO7/5uulVaj+NZzhf4yTw5qH83I7SvPG+RW14d+KB74xJiVBqse+VgAKpC RziQ== X-Received: by 10.60.31.206 with SMTP id c14mr43245351oei.88.1357899659506; Fri, 11 Jan 2013 02:20:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.60.31.206 with SMTP id c14mr43245347oei.88.1357899659379; Fri, 11 Jan 2013 02:20:59 -0800 (PST) Received: by 10.76.1.18 with HTTP; Fri, 11 Jan 2013 02:20:59 -0800 (PST) In-Reply-To: References: Date: Fri, 11 Jan 2013 15:50:59 +0530 Message-ID: Subject: Re: JobCache directory cleanup From: Hemanth Yamijala To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=e89a8ff252e0d682f704d300a620 X-Gm-Message-State: ALoCoQmvx51C3ETimjCwJdjO5yO6ZHa094kzRZSLiVJt9LZlenytifclG3oRvzychnBoJ80ermnY2UM3Gm6M9n/Gjy/H8g9z21w4d8RM2dBEVBvB4v8udyBAgT6qwdxBs55yDb3VV0ewdPbQJ1fgokkarOPdmnaXIw== X-Virus-Checked: Checked by ClamAV on apache.org --e89a8ff252e0d682f704d300a620 Content-Type: text/plain; charset=ISO-8859-1 Hmm. Unfortunately, there is another config variable that may be affecting this: keep.task.files.pattern This is set to .* in the job.xml file you sent. I suspect this may be causing a problem. Can you please remove this, assuming you have not set it intentionally ? Thanks Hemanth On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov wrote: > Thanks for replies! > > keep.failed.task.files set to false. > Config of one of the jobs attached. > > > On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala < > yhemanth@thoughtworks.com> wrote: > >> Good point. Forgot that one :-) >> >> >> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli < >> vinodkv@hortonworks.com> wrote: >> >>> >>> >>> Can you check the job configuration for these ~100 jobs? Do they have >>> keep.failed.task.files set to true? If so, these files won't be deleted. If >>> it doesn't, it could be a bug. >>> >>> Sharing your configs for these jobs will definitely help. >>> >>> Thanks, >>> +Vinod >>> >>> >>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov < >>> itretyakov@griddynamics.com> wrote: >>> >>>> Hello! >>>> >>>> I've found that jobcache directory became very large on our cluster, >>>> e.g.: >>>> >>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache >>>> 465G /data1/mapred/local/taskTracker/user/jobcache >>>> 464G /data2/mapred/local/taskTracker/user/jobcache >>>> 454G /data3/mapred/local/taskTracker/user/jobcache >>>> >>>> And it stores information for about 100 jobs: >>>> >>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/ | sort | >>>> uniq | wc -l >>>> >>> >> > > > -- > Best Regards > Ivan Tretyakov > > Deployment Engineer > Grid Dynamics > +7 812 640 38 76 > Skype: ivan.tretyakov > www.griddynamics.com > itretyakov@griddynamics.com > --e89a8ff252e0d682f704d300a620 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hmm. Unfortunately, there is another config variable that may be affecting = this: keep.task.files.pattern

This is set to .* in the j= ob.xml file you sent. I suspect this may be causing a problem. Can you plea= se remove this, assuming you have not set it intentionally ?

Thanks
Hemanth

=


On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <= itretyakov= @griddynamics.com> wrote:
Thanks for replies!

keep.= failed.task.files set to false.
Config of one of the jobs attache= d.


On Fri, Jan = 11, 2013 at 5:44 AM, Hemanth Yamijala <yhemanth@thoughtworks.com> wrote:
Good point. Forgot that one :-)


On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <vinodkv@h= ortonworks.com> wrote:


Can you check the job configuration = for these ~100 jobs? Do they have keep.failed.task.files set to true? If so= , these files won't be deleted. If it doesn't, it could be a bug.
Sharing your configs for these jobs will definitely help.

Thanks,
+Vinod


On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <itretyakov@griddynamics.com> wrote:
Hello!
I've found that jobcache directory became very large on our= cluster, e.g.:

# du -sh /data?/mapred/local/taskTracker/user/jobc= ache
465G =A0 =A0/data1/mapred/local/taskTracker/user/jobcache
464G =A0 =A0/data2/mapred/local/taskTracker/user/jobcache
45= 4G =A0 =A0/data3/mapred/local/taskTracker/user/jobcache

And it stores information for about 100 jobs:

#=A0ls -1 /data?/mapred/local/taskTracker/persona/jobcache/ =A0| sort | u= niq | wc -l




<= /div>
--
Best Regards
Ivan Tretyakov

Deployment=A0Engineer
Grid Dynamics
<= div> +7 812 640 38 76
Skype: ivan.tretyakov

--e89a8ff252e0d682f704d300a620--