hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1029) Allow embedding leader election into the RM
Date Sat, 14 Dec 2013 02:34:08 GMT

    [ https://issues.apache.org/jira/browse/YARN-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848168#comment-13848168
] 

Todd Lipcon commented on YARN-1029:
-----------------------------------

I agree with Karthik here -- the main reasons to pursue a separate ZKFC in HDFS were:
- avoid failover in the case of GC (since ZKFC has a very low heap requirement) but still
failover fast in machine failure.
- avoid adding any dependency on ZK within the NN
- allow the option to use other resource managers
-- in practice no one has done this and I think the extra complexity all of our pluggability
introduces is not worth it

In the case of RM HA, as I understand it (apologies if I got anything wrong - only tangentially
followed this discussion):
- RM HA uses ZK itself for shared storage, so it already has a dependency on ZK.
- Given that the shared state is in ZK, we don't need fencing if the same ZK client does election.
The reason is that, if an RM loses its ZK lease, it will simultaneously trigger the failover
_and_ be unable to make further changes in ZK. This exactly the semantics that we want.

Having a separate ZKFC actually complicates things, because we may have to reintroduce some
kind of fencing. What does it mean if the ZKFC loses its ZK lease, but the RM itself continues
to have access to ZK? It multiplies the 'state diagram' in two, and doesn't seem to offer
any particular advantages.

As for embedding ZKFC (and refactoring it so it can (a) not do health checks, (b) not control
the RM via RPC, but directly, (c) re-use the same ZK session) seems more complicated than
it's worth. Given we'd be throwing away all of the ZKFC features beyond the elector, why not
just use the elector?

I'm also not sure why we want to preserve the "external ZKFC" option - per above it's a more
complicated deployment scenario and seems to offer little tangible benefit.

> Allow embedding leader election into the RM
> -------------------------------------------
>
>                 Key: YARN-1029
>                 URL: https://issues.apache.org/jira/browse/YARN-1029
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Bikas Saha
>            Assignee: Karthik Kambatla
>         Attachments: embedded-zkfc-approach.patch, yarn-1029-0.patch, yarn-1029-0.patch,
yarn-1029-approach.patch
>
>
> It should be possible to embed common ActiveStandyElector into the RM such that ZooKeeper
based leader election and notification is in-built. In conjunction with a ZK state store,
this configuration will be a simple deployment option.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message