hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Clay" <rbc...@ncsu.edu>
Subject targeted balance operation
Date Thu, 24 Feb 2011 15:07:55 GMT
Hi-

 

We're working in an environment where we'd like to add a new DataNode and
quickly begin leveraging it, usually after detection of high load at the
rest of our DataNodes.  

 

However, most of the methods I can find regarding placing pre-existing
blocks on a new DataNode involve operations which are liable to perform
unnecessary IO at the established DataNodes, such as:

1.       Increasing replica count

2.       Adding and removing a file

3.       Global rebalance with a low threshold

 

Is there any capability currently built in to force unidirectional movement
of blocks to a targeted DataNode?  This would be ideal for us because we
already assume established DataNodes are under significant load, and we want
to begin using the new DataNode with as minimal inter-DataNode block traffic
as possible.

 

I wanted to confirm the lack of this capability with you folks before trying
to extend the Balancer class
(http://hadoop.apache.org/hdfs/docs/r0.21.0/api/org/apache/hadoop/hdfs/serve
r/balancer/Balancer.html).

 

Thanks!

 

-Ben

 


Mime
View raw message