hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Will <stefan.w...@gmx.net>
Subject Re: Datanode not detecting full disk
Date Wed, 29 Oct 2008 23:55:36 GMT
Hi Raghu,

Each DN machine has 3 partitions, e.g.:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              20G  8.0G   11G  44% /
/dev/sda3             1.4T  756G  508G  60% /data
tmpfs                 3.9G     0  3.9G   0% /dev/shm

All of the paths in hadoop-site.xml point to /data, which is the partition
that filled up to 100% (I deleted a bunch of files from HDFS since then). So
I guess the question is whether the DN looks at just the partition its data
directory is on, or all partitions when it determines disk usage.

-- Stefan


> From: Raghu Angadi <rangadi@YAHOO-INC.COM>
> Reply-To: <core-user@hadoop.apache.org>
> Date: Wed, 29 Oct 2008 11:57:07 -0700
> To: <core-user@hadoop.apache.org>
> Subject: Re: Datanode not detecting full disk
> 
> Stefan Will wrote:
>> Hi Jeff,
>> 
>> Yeah, it looks like I'm running into the issues described in the bug. I'm
>> running 0.18.1 on CentOS 5 by the way. Measuring available disk space
>> appears to be harder than I thought ... and here I was under the impression
>> the percentage in df was a pretty clear indicator of how full the disk is
>> ;-)
>> 
>> How did you guys solve/work around this ?
> 
> How many partitions do you have? If it is just one and NameNode thinks
> it has space though 'available' in df shows very less or no space, then
> you need to file a jira. There should be no case where DN reports more
> space than what 'available' field in 'df' shows.
> 
> But if you have more partitions and only some of them are full, then it
> is a different issue.. which should still be fixed.
> 
> Raghu.
> 
>> -- Stefan
>> 
>>  
>>> From: Jeff Hammerbacher <jeff.hammerbacher@gmail.com>
>>> Reply-To: <core-user@hadoop.apache.org>
>>> Date: Mon, 27 Oct 2008 12:40:08 -0700
>>> To: <core-user@hadoop.apache.org>
>>> Subject: Re: Datanode not detecting full disk
>>> 
>>> Hey Stefan,
>>> 
>>> We used to have trouble with this issue at Facebook. What version are
>>> you running? You might get more information on this ticket:
>>> https://issues.apache.org/jira/browse/HADOOP-2991.
>>> 
>>> Regards,
>>> Jeff
>>> 
>>> On Mon, Oct 27, 2008 at 10:00 AM, Stefan Will <stefan.will@gmx.net> wrote:
>>>> Each of my datanodes has  a system and a data partition, with dfs.data.dir
>>>> pointed to the data partition. The data partition just filled up to 100%
on
>>>> all of my nodes (as evident via df), but the NameNode web ui still shows
>>>> them only 88-94% full (interestingly, the numbers differ even though the
>>>> machines are configured identically). I thought the datanodes used df to
>>>> determine free space ? How is the storage utilization determined ?
>>>> 
>>>> -- Stefan
>>>> 
>> 
>> 



Mime
View raw message