hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: A question of QJM with HDFS federation
Date Mon, 08 Apr 2013 06:50:34 GMT
Hi Azurry,


Yes, multiple nameservices can share a single QJM set. The QJM
configuration allows for a journal ID prefix path which you should
configure to be the nameservice ID. You do not need to change disk
paths/etc. at all.

For example,

NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:

NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:

Which will separate both of them logically and still make them use the
same QJM set of nodes.


Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
the same ha.zookeeper.quorum value since the ZKFC automatically reuses
the nameservice ID as its parent znode name on the ZK instance, and
won't collide with another NSs' ZKFCs.

Do post back if there are still some more doubts.

On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <azuryyyu@gmail.com> wrote:
> Hi dears,
> I deployed Hadoopv2 with HA enabled using QJM, so my question is:
> 1) if we also configured HDFS federation,  such as:
>     NN1 is active, NN2 is standby
>     NN3 is active, NN4 is standby
> they are configured as HDFS federation, then,
> Can these four NNs using the same Journal nodes and ZKs?
> if your answer is yes, does that enough just using different
> dfs.journalnode.edits.dir?
> such as:
> NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
> Thanks.

Harsh J

View raw message