hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
Date Wed, 11 Jun 2014 10:24:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14027601#comment-14027601

Rakesh R commented on HADOOP-10641:

bq.The intent of this jira is not to solve the general problem of distributed consensus. That
is, I do not propose to build an implementation of paxos or other coordination algorithms
here. This is only to introduce a common interface, so that real implementations such as ZooKeeper
could be plugged into hadoop projects.

This sounds interesting. Thanks for the effort!. If I understood the discussion correctly
here, the idea is to build a quorum based replication. For example, the events(I think this
represents data) are submitted as proposals to a quorum of nodes. In ZooKeeper terms, Leader
proposes values to the Followers. Now Leader wait for acknowledgements from a quorum of Followers
before considering a proposal committed. Also, Leader queues COMMIT(zxid) events to all Followers
so that all other nodes learn the events. This ensures that the events will be reached to
all nodes in the system. Adding one more point, in general ZK provides strong ordering guarantees.

Sometime back ZooKeeper folks initiated discussions to decouple ZAB from ZooKeeper, so that
users can make use of this and can define their own models and reliably replicate the data.
There is a related JIRA ZOOKEEPER-1931 talks similar feature, now this is in initial dev stage.
Please have a look at this. I hope this would help to define a common interface, also an opportunity
for us to know more about the use cases.


> Introduce Coordination Engine
> -----------------------------
>                 Key: HADOOP-10641
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10641
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 3.0.0
>            Reporter: Konstantin Shvachko
>            Assignee: Plamen Jeliazkov
>         Attachments: HADOOP-10641.patch, HADOOP-10641.patch, HADOOP-10641.patch
> Coordination Engine (CE) is a system, which allows to agree on a sequence of events in
a distributed system. In order to be reliable CE should be distributed by itself.
> Coordination Engine can be based on different algorithms (paxos, raft, 2PC, zab) and
have different implementations, depending on use cases, reliability, availability, and performance
> CE should have a common API, so that it could serve as a pluggable component in different
projects. The immediate beneficiaries are HDFS (HDFS-6469) and HBase (HBASE-10909).
> First implementation is proposed to be based on ZooKeeper.

This message was sent by Atlassian JIRA

View raw message