hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From LAT <tome...@gmail.com>
Subject HDFS Federation and multiple clusters
Date Thu, 12 Sep 2013 15:03:48 GMT
Reading the HDFS Federation documentation, it seems that it provides
support for multiple NameNodes in a single cluster.  The DataNodes are
shared across all NameNodes.  It seems one can take a single hadoop
cluster, and add HDFS Federation, but I do not see any way to take multiple
hadoop clusters and make the composite cluster federated; and then,
undo/change the definition of the composite cluster back to individual
clusters as needed.

Am I reading this correctly? Or is there a way to define a federated hadoop
cluster from multiple clusters?  It seems the DataNodes would need to be
separate by original cluster to accomplish this.

My interest stems from a desire to allow a compute service to issue
map/reduce jobs to the multiple hadoop cluster, without say needing
internally managed catalogs, etc.  Ideally the composite hadoop cluster
could be dynamically changed to include those hadoop clusters of interest,
with the ease analogous to changing /etc/fstab.

Appreciate any thoughts on this.


View raw message