hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shangan" <shan...@corp.kaixin001.com>
Subject Re: Re: how to make hadoop balance automatically
Date Fri, 08 Oct 2010 05:39:31 GMT
is there any way to change the default storage policy ? for example: don't store the first
copy of a block on the local node but distribute the copies randomly instread 


发件人: Raj V 
发送时间: 2010-09-28  22:28:12 
收件人: common-user 
主题: Re: how to make hadoop balance automatically 
The first copy of a block is always stored on the local node. If you want a 
balanced distribution, do the data moving from  the name node and don't  make 
the name node into a data node.
From: Neil Xu <neil.xuxf@gmail.com>
To: common-user@hadoop.apache.org
Sent: Tue, September 28, 2010 3:13:01 AM
Subject: Re: how to make hadoop balance automatically
Hi, Shangan
you can find something useful at
and the document
shows how to rebalance.
I think you can try to set more mappers (much larger than the number of
nodes), and see if it will be improved.
在 2010年9月28日 下午4:09,shangan <shangan@corp.kaixin001.com>写道:
> I have a cluster of 30 nodes, and I put data into the cluster on one node I
> called "NodeA" here. The consequence is that now this node always stores
> more data than other node, for example other nodes store 10G to 15G,while
> NodeA will store 50G to 60G .
> do anyone know what cause such consequence  and how to avoid it ?
> btw: I know there a balancer tool can do balance
> 2010-09-28
> shangan
__________ Information from ESET NOD32 Antivirus, version of virus signature database 5484
(20100927) __________
The message was checked by ESET NOD32 Antivirus.
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message