hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@veoh.com>
Subject Re: Limit the space used by hadoop on a slave node
Date Tue, 08 Jan 2008 22:37:07 GMT

I don't have the specific data you request, but I can give you a general
outline for the dev cluster in question.

I have 4 nodes that are general use.  These have about 1TB of storage each,
but this is largely used by other processes.  These nodes usually have
50-500GB free.

I have 8 nodes that have one 70GB drive and one 500GB drive.  The 70GB drive
usually has about 40GB free.  The 500GB drive is essentially all for hadoop.

I have 2 nodes that have one 70GB drive that usually has about 40GB of free
space.

Originally, my storage partitions were listed as
small-partition,large-partition.  I later changed that to
large-partition,small-partition and then changed it again to list only the
partition available on the machine.  A utility to evacuate a partition would
come in very handy here, btw.  Turning off one node at a time and waiting
for the blocks to replicate is very slow.  It would be much nicer to be able
to announce to hadoop that I want the blocks on a particular disk partition
re-replicated NOW.  Since I had 8 partitions to evacuate and some of these
had been slightly corrupted due to disk-full conditions, evacuating them
took forever.  

I would have loved to have been able to just say that those partitions
should not be counted as replicants (but should be considered as possible
replication sources).  I would also have appreciated some way to tell the
cluster to prioritize replication of blocks at risk ahead of normal
computation.  This is especially important if somebody is running with only
2 copies of files.  Fsck should also have an option to cause it to trigger
block reports from data nodes so that latent problems can be flushed out of
hiding.

I had about 20% usage across available storage.


On 1/8/08 2:16 PM, "Hairong Kuang" <hairong@yahoo-inc.com> wrote:

> I agree that block distribution does not deal with heterogeneous cluster
> well. Basically block replication does not favor less utilized datanode.
> After 0.16 is released, you may periodically run the balancer to
> redistribute blocks with the command bin/start-balancer.sh.
> 
> I checked the datanode code. A datanode does check the amount of
> available space before block allocation. I need to investigate the cause
> of the disk full problem. I appreciate if you could provide me more
> information like the capacity of the disk, the amount of dfs used space,
> reserved space, and non-dfs used space when the out of disk problem
> occurs.
> 
> Hairong
> 
> -----Original Message-----
> From: Ted Dunning [mailto:tdunning@veoh.com]
> Sent: Tuesday, January 08, 2008 1:37 PM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Limit the space used by hadoop on a slave node
> 
> 
> And I have both but have had disk full problems.  I can't be sure right
> now whether this occurred under 14.4 or 15.1, but I think it was 15.1.
> 
> In any case, new file creation from a non-datanode host is definitely
> not well balanced and will lead to disk full conditions if you have
> dramatically different sized partitions available on the different
> datanodes.  Also, if you have a small and a large partition available on
> a single node, the small partition will fill up and cause corruption.  I
> had to go to single partitions on all nodes to avoid this.
> 
> <property>
>   <name>dfs.datanode.du.reserved</name>
>   <!--  10 GB -->
>   <value> 10000000000 </value>
>   <description>Reserved space in bytes. Always leave this much space
> free for non dfs use  </description> </property>
> 
> <property>
>   <name>dfs.datanode.du.pct</name>
>   <value>0.9f</value>
>   <description>When calculating remaining space, only use this
> percentage of the real available space
>   </description>
> </property>
> 
> 
> 
> On 1/8/08 1:30 PM, "Koji Noguchi" <knoguchi@yahoo-inc.com> wrote:
> 
>> We use,
>> 
>> dfs.datanode.du.pct for 0.14 and dfs.datanode.du.reserved for 0.15.
>> 
>> Change was made in the Jira Hairong mentioned.
>> https://issues.apache.org/jira/browse/HADOOP-1463
>> 
>> Koji
>> 
>>> -----Original Message-----
>>> From: Ted Dunning [mailto:tdunning@veoh.com]
>>> Sent: Tuesday, January 08, 2008 1:13 PM
>>> To: hadoop-user@lucene.apache.org
>>> Subject: Re: Limit the space used by hadoop on a slave node
>>> 
>>> 
>>> I think I have seen related bad behavior on 15.1.
>>> 
>>> On 1/8/08 11:49 AM, "Hairong Kuang" <hairong@yahoo-inc.com> wrote:
>>> 
>>>> Has anybody tried 15.0? Please check
>>>> https://issues.apache.org/jira/browse/HADOOP-1463.
>>>> 
>>>> Hairong
>>>> -----Original Message-----
>>>> From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]
>>>> Sent: Tuesday, January 08, 2008 11:33 AM
>>>> To: hadoop-user@lucene.apache.org; hadoop-user@lucene.apache.org
>>>> Subject: RE: Limit the space used by hadoop on a slave node
>>>> 
>>>> at least up until 14.4, these options are broken. see
>>>> https://issues.apache.org/jira/browse/HADOOP-2549
>>>> 
>>>> (there's a trivial patch - but i am still testing).
>>>> 
>>>> 
>> 
> 


Mime
View raw message