hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayuran Yogarajah <mayuran.yogara...@casalemedia.com>
Subject Re: NN + secondary got full, even though data nodes had plenty of space
Date Wed, 12 Aug 2009 17:01:44 GMT
Todd Lipcon wrote:
> Hi Mayuran,
>
> Do you do all of your uploads of data into your Hadoop cluster from node001
> and node002?
>
> If so, keep in mind that one of your replicas will always be written on
> localhost in the case that it is part of the cluster.
>
> You should consider running the rebalancer to even up your space usage.
>
> -Todd
>
>   
Actually yes I have been doing this.  I'll try rebalancer, thanks for 
your help.

M


> On Tue, Aug 11, 2009 at 11:09 AM, Mayuran Yogarajah <
> mayuran.yogarajah@casalemedia.com> wrote:
>
>   
>> I have a 6 node cluster running Hadoop 0.18.3.  I'm trying to figure out
>> how the data was spread out like this:
>>
>> node001         94.15%
>> node002         94.16%
>> node003         48.22%
>> node004         47.85%
>> node005         48.12%
>> node006         43.18%
>> Node 001 (NN) and node 002( secondary NN) both got full, while the other
>> data nodes had more space left.  I had assumed that Hadoop would distribute
>> more blocks to nodes 3-6 since they had much more space, but it ended up
>> filling up nodes1 and 2.  Is this expected?
>>
>> thanks,
>> M
>>
>>
>>     


Mime
View raw message