hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexandra Alecu <alexandra.al...@gmail.com>
Subject How to replace the storage on a datanode without formatting the namenode?
Date Thu, 14 May 2009 14:44:07 GMT


I want to test how Hadoop and HBase are performing. I have a cluster with 1
namenode and 4 datanodes. I use Hadoop 0.19.1 and HBase 0.19.2. 

I first ran a few tests when the 4 datanodes use local storage specified in
Now, I want to see what is the tradeoff if I switch from local storage to
network mounted storage (I know it sounds like a crazy idea but
unfortunately I have to explore this possibility).

I would like to be able to change the dfs.data.dir and maybe in two steps be
able to switch to the network mounted storage.

What I had in mind was the following steps : 

0. Assume initial status is a working cluster with local storage, e.g.
dfs.data.dir set to local_storage_path.
1. Stop cluster: bin/stop-dfs
2. Change dfs.data.dir by adding the network_storage_path to the local
3. Start cluster: bin/start-dfs (this will format the new network locations,
which is nice)
4. <Perform some sort of directed balancing of all the data towards the
network storage location>
5. Stop cluster: bin/stop-dfs
6. Change dfs.data.dir parameter to only contain local_storage_path
7.  Start cluster and live happily ever after :-).

The problem is , I don;t know if there is a command or an option to achieve
step 4.
Do you have any suggestions ?

I found some info on how to add datanodes, but there is not much info on how
to remove safely (without losing data etc) datanodes or storage locations on
a particular node.
Is this possible? 

Many thanks,

View this message in context: http://www.nabble.com/How-to-replace-the-storage-on-a-datanode-without-formatting-the-namenode--tp23542127p23542127.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

View raw message