hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan A. P. Pendleton" ...@geekdom.net>
Subject Re: Some queries on Master node
Date Wed, 01 Nov 2006 23:55:24 GMT
>
> > 1. Will Hadoop move the blocks in master node to the newly added nodes?
>
> No, HDFS will not move blocks to new nodes automatically. New files that
> are added will likely have their blocks placed on the new nodes. Also,
> removing old files will remove some blocks that are on your older nodes.
>
> One way to re-balance your cluster would be to:
> - Select a subset of files that take up a good percentage of your disk
> space
> - Copy them to new locations in your HDFS
> - Remove the *old* copies of the files
> - Rename the new copies to their previous names


A simpler way, with no interruption of service, is to merely turn up the
replication of files, wait for transfers to stabilize, then turn the
replication back down.

This will tend to distribute data from "full" nodes to "empty" nodes. And,
it can be done to any subset of your files, as long as there is some room to
add extra replications of each file you're up-replicating.

Of course, it'd still be lovely to have some background I/O dedicated to
space re-balancing, but this is a simple-enough way to get things
redistributed.

-- 
Bryan A. P. Pendleton
Ph: (877) geek-1-bp

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message