Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <4A82F578.7050309@casalemedia.com>
Date: Wed, 12 Aug 2009 13:01:44 -0400
From: Mayuran Yogarajah <mayuran.yogarajah@casalemedia.com>
User-Agent: Thunderbird 2.0.0.22 (X11/20090605)
MIME-Version: 1.0
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Subject: Re: NN + secondary got full, even though data nodes had plenty of
 	space
References: <4A81B3C3.3060903@casalemedia.com>
 <45f85f70908120823h1dc26130ncd6ce1b700035f1a@mail.gmail.com>
In-Reply-To: <45f85f70908120823h1dc26130ncd6ce1b700035f1a@mail.gmail.com>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit

Todd Lipcon wrote:
> Hi Mayuran,
>
> Do you do all of your uploads of data into your Hadoop cluster from node001
> and node002?
>
> If so, keep in mind that one of your replicas will always be written on
> localhost in the case that it is part of the cluster.
>
> You should consider running the rebalancer to even up your space usage.
>
> -Todd
>
>   
Actually yes I have been doing this.  I'll try rebalancer, thanks for 
your help.

M


> On Tue, Aug 11, 2009 at 11:09 AM, Mayuran Yogarajah <
> mayuran.yogarajah@casalemedia.com> wrote:
>
>   
>> I have a 6 node cluster running Hadoop 0.18.3.  I'm trying to figure out
>> how the data was spread out like this:
>>
>> node001         94.15%
>> node002         94.16%
>> node003         48.22%
>> node004         47.85%
>> node005         48.12%
>> node006         43.18%
>> Node 001 (NN) and node 002( secondary NN) both got full, while the other
>> data nodes had more space left.  I had assumed that Hadoop would distribute
>> more blocks to nodes 3-6 since they had much more space, but it ended up
>> filling up nodes1 and 2.  Is this expected?
>>
>> thanks,
>> M
>>
>>
>>