Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D9A76F1D5 for ; Thu, 11 Apr 2013 07:29:20 +0000 (UTC) Received: (qmail 33395 invoked by uid 500); 11 Apr 2013 07:29:15 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 33009 invoked by uid 500); 11 Apr 2013 07:29:11 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 32982 invoked by uid 99); 11 Apr 2013 07:29:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Apr 2013 07:29:10 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hemanty@thoughtworks.com designates 64.18.0.145 as permitted sender) Received: from [64.18.0.145] (HELO exprod5og103.obsmtp.com) (64.18.0.145) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Apr 2013 07:29:05 +0000 Received: from mail-ve0-f198.google.com ([209.85.128.198]) (using TLSv1) by exprod5ob103.postini.com ([64.18.4.12]) with SMTP ID DSNKUWZmLGFbP6ibN11wMHUMoeTi5YhjjoUY@postini.com; Thu, 11 Apr 2013 00:28:44 PDT Received: by mail-ve0-f198.google.com with SMTP id ox1so2204648veb.5 for ; Thu, 11 Apr 2013 00:28:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-received:in-reply-to:references:date :message-id:subject:from:to:content-type:x-gm-message-state; bh=8Bl6jRxLHVyxrLuw2GFKDNogNzmhwbGEEiTXL//ZSUc=; b=WeucC5ABu0MaKPsKVO7VaBrHdSlA6BmSXXSGOVd5UHM9fVoBfemFEzJIDPDZSkaLCD eJ4V83GbkzAjpVcWOytFlUlJ8L0eQbr6jbUBKo+iJeV0bscor5pHR9SdKa8ZhccQJzHm xdCqPpwME4zYVsieSmy0vvaGb9vn+JWMul/vW5wEVYj96PxINvK8pXl8DZ9fQyPyrVqg CpELTgZ3z0HBkDa6PU8hRoxyZ+Q1lIdFBGA1LchHNFuwna9oELYRWEUgwXID+C4a6zQO Q6tjpWna+QrX2yAa6hamyxqvpbY226A1GhHXDxcnvPrSJFbxdh5KelOnBdC/ER5N85cr 6kwg== X-Received: by 10.52.177.229 with SMTP id ct5mr3472768vdc.113.1365665323874; Thu, 11 Apr 2013 00:28:43 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.52.177.229 with SMTP id ct5mr3472756vdc.113.1365665323672; Thu, 11 Apr 2013 00:28:43 -0700 (PDT) Received: by 10.220.220.13 with HTTP; Thu, 11 Apr 2013 00:28:43 -0700 (PDT) In-Reply-To: References: <2D2295EC-2713-47C4-90B4-E16BA6467954@hortonworks.com> Date: Thu, 11 Apr 2013 12:58:43 +0530 Message-ID: Subject: Re: How to configure mapreduce archive size? From: Hemanth Yamijala To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=20cf3071cd327ff54404da10bcfa X-Gm-Message-State: ALoCoQlxcnbn+Ih+t7gByOEkuy18e+ol59wAHO+5kKW6r8PyrtrIUvIYq3yCGKBHHtvt3uubmCjzv8FoFL3OP9Sremf7LOleaLs4+InTs/4hOtq3bW05VS9aoklciuGQCy8Jp0qDR+H1oBVwFCtiOEf2dUD8PThfhg== X-Virus-Checked: Checked by ClamAV on apache.org --20cf3071cd327ff54404da10bcfa Content-Type: text/plain; charset=ISO-8859-1 Could you paste the contents of the directory ? Not sure whether that will help, but just giving it a shot. What application are you using ? Is it custom MapReduce jobs in which you use Distributed cache (I guess not) ? Thanks Hemanth On Thu, Apr 11, 2013 at 3:34 AM, wrote: > Hi Arun,**** > > ** ** > > I stopped my application, then restarted my hbase (which include hadoop). > After that I start my application. After one evening, my /tmp/hadoop-root/mapred/local/archive > goes to more than 1G. It does not work.**** > > ** ** > > Is this the right place to change the value?**** > > ** ** > > "local.cache.size" in file core-default.xml, which is in > hadoop-core-1.0.3.jar**** > > ** ** > > Thanks,**** > > ** ** > > Jane**** > > ** ** > > *From:* Arun C Murthy [mailto:acm@hortonworks.com] > *Sent:* Wednesday, April 10, 2013 2:45 PM > > *To:* user@hadoop.apache.org > *Subject:* Re: How to configure mapreduce archive size?**** > > ** ** > > Ensure no jobs are running (cache limit is only for non-active cache > files), check after a little while (takes sometime for the cleaner thread > to kick in).**** > > ** ** > > Arun**** > > ** ** > > On Apr 11, 2013, at 2:29 AM, > wrote:**** > > > > **** > > Hi Hemanth,**** > > **** > > For the hadoop 1.0.3, I can only find "local.cache.size" in file > core-default.xml, which is in hadoop-core-1.0.3.jar. It is not in > mapred-default.xml.**** > > **** > > I updated the value in file default.xml and changed the value to 500000. > This is just for my testing purpose. However, the folder > /tmp/hadoop-root/mapred/local/archive already goes more than 1G now. Looks > like it does not do the work. Could you advise if what I did is correct?** > ** > > **** > > local.cache.size**** > > 500000**** > > **** > > Thanks,**** > > **** > > Xia**** > > **** > > *From:* Hemanth Yamijala [mailto:yhemanth@thoughtworks.com] > *Sent:* Monday, April 08, 2013 9:09 PM > *To:* user@hadoop.apache.org > *Subject:* Re: How to configure mapreduce archive size?**** > > **** > > Hi,**** > > **** > > This directory is used as part of the 'DistributedCache' feature. ( > http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html#DistributedCache). > There is a configuration key "local.cache.size" which controls the amount > of data stored under DistributedCache. The default limit is 10GB. However, > the files under this cannot be deleted if they are being used. Also, some > frameworks on Hadoop could be using DistributedCache transparently to you. > **** > > **** > > So you could check what is being stored here and based on that lower the > limit of the cache size if you feel that will help. The property needs to > be set in mapred-default.xml.**** > > **** > > Thanks**** > > Hemanth**** > > **** > > On Mon, Apr 8, 2013 at 11:09 PM, wrote:**** > > Hi,**** > > **** > > I am using hadoop which is packaged within hbase -0.94.1. It is hadoop > 1.0.3. There is some mapreduce job running on my server. After some time, I > found that my folder /tmp/hadoop-root/mapred/local/archive has 14G size.** > ** > > **** > > How to configure this and limit the size? I do not want to waste my space > for archive.**** > > **** > > Thanks,**** > > **** > > Xia**** > > **** > > **** > > ** ** > > --**** > > Arun C. Murthy**** > > Hortonworks Inc. > http://hortonworks.com/**** > > ** ** > --20cf3071cd327ff54404da10bcfa Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Could you paste the contents of the directory ? Not sure w= hether that will help, but just giving it a shot.

= What application are you using ? Is it custom MapReduce jobs in which you u= se Distributed cache (I guess not) ?=A0

Thanks
Hemanth
<= div class=3D"gmail_extra">

On Thu, Apr 11= , 2013 at 3:34 AM, <Xia_Yang@dell.com> wrote:

Hi Arun,

=A0<= /p>

I stopped my applicati= on, then restarted my hbase (which include hadoop). After that I start my a= pplication. After one evening, my /tmp/ha= doop-root/mapred/local/archive goes to more than 1G. It does not work.

=A0<= /p>

Is this the right plac= e to change the value?

= =A0

&quo= t;local.cache.size" in file core-default.xml, which is in hadoop-core-= 1.0.3.jar

=A0<= /p>

Thanks,

=A0<= /p>

Jane

=A0<= /p>

From: Arun C M= urthy [mailto:acm@= hortonworks.com]
Sent: Wednesday, April 10, 2013 2:45 PM


To: user@hadoop.apache.org
Subject: Re: How to configure map= reduce archive size?

<= /u>=A0

Ensure no jobs are running (cache l= imit is only for non-active cache files), check after a little while (takes= sometime for the cleaner thread to kick in).

=A0

Arun

=A0<= u>

On Apr 11, 2013, at 2:29 AM, <= ;Xia_Yang@Dell.com> <Xia_Yang@= Dell.com> wrote:



Hi Hemanth,<= /p>

=A0=

For th= e hadoop 1.0.3, I can only find "local.cache.size" in file core-d= efault.xml, which is in hadoop-core-1.0.3.jar. It is not in mapred-default.= xml.

=A0<= /u>

= I updated the value in file default.xml and changed the value to 500000. Th= is is just for my testing purpose. However, the folder /tmp/hadoop-root/map= red/local/archive already goes more than 1G now. Looks like it does not do = the work. Could you advise if what I did is correct?

=A0<= /u>

= =A0 <name>local.cache.size</name>

=A0 <value&= gt;500000</value>

=A0

Thanks,

=A0<= /u>

= Xia

=A0<= /u>

From:= =A0Hemanth Yamijala [mailto= :yhemanth@th= oughtworks.com]=A0
Sent:=A0Monday, April 08, 2013 9:09 PM
To:=A0us= er@hadoop.apache.org
Subject:=A0Re: How to confi= gure mapreduce archive size?

=A0

Hi,

=A0

This = directory is used as part of the 'DistributedCache' feature. (http://hadoop.apache.org/docs/r1.0.4/mapred_tutori= al.html#DistributedCache). There is a configuration key "local.cac= he.size" which controls the amount of data stored under DistributedCac= he. The default limit is 10GB. However, the files under this cannot be dele= ted if they are being used. Also, some frameworks on Hadoop could be using = DistributedCache transparently to you.

=A0

So you could check what is being stored= here and based on that lower the limit of the cache size if you feel that = will help. The property needs to be set in mapred-default.xml.

=A0

Thanks

Hemanth

=A0

On Mon, Apr 8, 2013 a= t 11:09 PM, <Xia_= Yang@dell.com> wrote:

Hi,

=A0

I am using hadoop which is packaged with= in hbase -0.94.1. It is hadoop 1.0.3. There is some mapreduce job running o= n my server. After some time, I found that my folder /tmp/hadoop-root/mapre= d/local/archive has 14G size.

=A0

How to configure this and limit the size? I do not want=A0 t= o waste my space for archive.

=A0

Thanks,

=A0

Xi= a

=A0

<= /div>

=A0

=A0

--

Arun C. Murthy

Hortonworks Inc.
http://hortonworks.co= m/

= =A0


--20cf3071cd327ff54404da10bcfa--