hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hairong Kuang <hair...@yahoo-inc.com>
Subject Re: Does Hadoop Honor Reserved Space?
Date Tue, 11 Mar 2008 00:16:56 GMT
I think you have a misunderstanding of the reserved parameter. As I
commented on hadoop-1463, remember that dfs.du.reserve is the space for
non-dfs usage, including the space for map/reduce, other application, fs
meta-data etc. In your case since /usr already takes 45GB, it far exceeds
the reserved limit 1G. You should set the reserved space to be 50G.

Hairong


On 3/10/08 4:54 PM, "Joydeep Sen Sarma" <jssarma@facebook.com> wrote:

> Filed https://issues.apache.org/jira/browse/HADOOP-2991
> 
> -----Original Message-----
> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
> Sent: Monday, March 10, 2008 12:56 PM
> To: core-user@hadoop.apache.org; core-user@hadoop.apache.org
> Cc: Pete Wyckoff
> Subject: RE: Does Hadoop Honor Reserved Space?
> 
> folks - Jimmy is right - as we have unfortunately hit it as well:
> 
> https://issues.apache.org/jira/browse/HADOOP-1463 caused a regression.
> we have left some comments on the bug - but can't reopen it.
> 
> this is going to be affecting all 0.15 and 0.16 deployments!
> 
> 
> -----Original Message-----
> From: Hairong Kuang [mailto:hairong@yahoo-inc.com]
> Sent: Thu 3/6/2008 2:01 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Does Hadoop Honor Reserved Space?
>  
> In addition to the version, could you please send us a copy of the
> datanode
> report by running the command bin/hadoop dfsadmin -report?
> 
> Thanks,
> Hairong
> 
> 
> On 3/6/08 11:56 AM, "Joydeep Sen Sarma" <jssarma@facebook.com> wrote:
> 
>> but intermediate data is stored in a different directory from dfs/data
>> (something like mapred/local by default i think).
>> 
>> what version are u running?
>> 
>> 
>> -----Original Message-----
>> From: Ashwinder Ahluwalia on behalf of ahluwalia5@yahoo.com
>> Sent: Thu 3/6/2008 10:14 AM
>> To: core-user@hadoop.apache.org
>> Subject: RE: Does Hadoop Honor Reserved Space?
>>  
>> I've run into a similar issue in the past. From what I understand,
> this
>> parameter only controls the HDFS space usage. However, the
> intermediate data
>> in
>> the map reduce job is stored on the local file system (not HDFS) and
> is not
>> subject to this configuration.
>> 
>> In the past I have used mapred.local.dir.minspacekill and
>> mapred.local.dir.minspacestart to control the amount of space that is
>> allowable
>> for use by this temporary data.
>> 
>> Not sure if that is the best approach though, so I'd love to hear what
> other
>> people have done. In your case, you have a map-red job that will
> consume too
>> much space (without setting a limit, you didn't have enough disk
> capacity for
>> the job), so looking at mapred.output.compress and
> mapred.compress.map.output
>> might be useful to decrease the job's disk requirements.
>> 
>> --Ash
>> 
>> -----Original Message-----
>> From: Jimmy Wan [mailto:jimmy@indeed.com]
>> Sent: Thursday, March 06, 2008 9:56 AM
>> To: core-user@hadoop.apache.org
>> Subject: Does Hadoop Honor Reserved Space?
>> 
>> I've got 2 datanodes setup with the following configuration parameter:
>> <property>
>>  <name>dfs.datanode.du.reserved</name>
>>  <value>429496729600</value>
>>  <description>Reserved space in bytes per volume. Always leave this
>> much  
>> space free for non dfs use.
>>  </description>
>> </property>
>> 
>> Both are housed on 800GB volumes, so I thought this would keep about
> half
>> the volume free for non-HDFS usage.
>> 
>> After some long running jobs last night, both disk volumes were
> completely
>> filled. The bulk of the data was in:
>> ${my.hadoop.tmp.dir}/hadoop-hadoop/dfs/data
>> 
>> This is running as the user hadoop.
>> 
>> Am I interpretting these parameters incorrectly?
>> 
>> I noticed this issue, but it is marked as closed:
>> http://issues.apache.org/jira/browse/HADOOP-2549
> 
> 
> 


Mime
View raw message