hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: How to configure mapreduce archive size?
Date Wed, 10 Apr 2013 21:44:48 GMT
Ensure no jobs are running (cache limit is only for non-active cache files), check after a
little while (takes sometime for the cleaner thread to kick in).

Arun

On Apr 11, 2013, at 2:29 AM, <Xia_Yang@Dell.com> <Xia_Yang@Dell.com> wrote:

> Hi Hemanth,
>  
> For the hadoop 1.0.3, I can only find "local.cache.size" in file core-default.xml, which
is in hadoop-core-1.0.3.jar. It is not in mapred-default.xml.
>  
> I updated the value in file default.xml and changed the value to 500000. This is just
for my testing purpose. However, the folder /tmp/hadoop-root/mapred/local/archive already
goes more than 1G now. Looks like it does not do the work. Could you advise if what I did
is correct?
>  
>   <name>local.cache.size</name>
>   <value>500000</value>
>  
> Thanks,
>  
> Xia
>  
> From: Hemanth Yamijala [mailto:yhemanth@thoughtworks.com] 
> Sent: Monday, April 08, 2013 9:09 PM
> To: user@hadoop.apache.org
> Subject: Re: How to configure mapreduce archive size?
>  
> Hi,
>  
> This directory is used as part of the 'DistributedCache' feature. (http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html#DistributedCache).
There is a configuration key "local.cache.size" which controls the amount of data stored under
DistributedCache. The default limit is 10GB. However, the files under this cannot be deleted
if they are being used. Also, some frameworks on Hadoop could be using DistributedCache transparently
to you.
>  
> So you could check what is being stored here and based on that lower the limit of the
cache size if you feel that will help. The property needs to be set in mapred-default.xml.
>  
> Thanks
> Hemanth
>  
> 
> On Mon, Apr 8, 2013 at 11:09 PM, <Xia_Yang@dell.com> wrote:
> Hi,
>  
> I am using hadoop which is packaged within hbase -0.94.1. It is hadoop 1.0.3. There is
some mapreduce job running on my server. After some time, I found that my folder /tmp/hadoop-root/mapred/local/archive
has 14G size.
>  
> How to configure this and limit the size? I do not want  to waste my space for archive.
>  
> Thanks,
>  
> Xia
>  
>  

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



Mime
View raw message