hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: Physically moving HDFS cluster to new
Date Thu, 18 Apr 2013 02:23:10 GMT
It may or may not help you in your current distress, but MapR's
distribution could handle this pretty easily.

One method is direct distcp between clusters, but you could also use MapR's
mirroring capabilities to migrate data.

You can also carry a MapR cluster, change the IP addresses and relight the
cluster without data loss.  You can also move disks (respecting RAID-0 disk
groups, of course) from machine to machine within a cluster and have them
wake up with all file and directory meta-data intact.

Furthermore, you can lose any two machines in a cluster and are guaranteed
to be able to reconstruct the cluster.  Even if you lose all three replicas
of *any* of the data or meta-data in the cluster, you can *still*
reconstruct any data volumes for which at least one copy survives.  The the
lost data volumes come back at a later time, you will also be able
resurrect the data correctly.

None of this is true for any of the other major Hadoop distributions.

Let me know if you want to try this out.

On Wed, Apr 17, 2013 at 5:20 PM, Tom Brown <tombrown52@gmail.com> wrote:

> We have a situation where we want to physically move our small (4 node)
> cluster from one data center to another. As part of this move, each node
> will receive both a new FQN and a new IP address. As I understand it, HDFS
> is somehow tied to the the FQN or IP address, and changing them causes data
> loss.
> Is there any supported method of moving a cluster this way?
> Thanks in advance!
> --Tom

View raw message