hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
Date Fri, 11 Jul 2014 18:16:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14059130#comment-14059130

Konstantin Shvachko commented on HADOOP-10641:

Sorry if it sounded as an overreaction, non intended.

??doing this work on a branch??
There is no resistance. The plan has always been to build CNode on a branch. I am just trying
to optimize development of CNode and HBase region replication, which is going on in parallel.
My thinking was to commit the CE interface to trunk and then branch off HDFS of it. That way
both both HDFS and HBase can use the interface.

??the primary goal of this work is to introduce a plugin point for WANdisco's coordination
engine implementation??
I don't see anything bad with plugging in WANdisco CE into Hadoop, [as I argued in the other
jira comment|https://issues.apache.org/jira/browse/HDFS-6469?focusedCommentId=14057235&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14057235].
But saying its a primary goal is not fair, you know me better than that.

Let me comment on the design. We actually looked at multiple consensus algorithms and their
implementations and came up with an abstractions that suite the area in the most general way.
Particularly, the call back from agreement to update the application state is separated from
the proposing action is because it is more generic. With some implementations of Raft a proposer
can just wait when the agreement is made and then proceed with its execution - synchronously.
But with ZK you have to set a watcher and wait for a callback acknowledging the event - asynchronously.
So Asynchronous approach wins as more generic.

If you take Bart we can organize pickup for participants from the near station. Also we should
have a dial up.

> Introduce Coordination Engine
> -----------------------------
>                 Key: HADOOP-10641
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10641
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 3.0.0
>            Reporter: Konstantin Shvachko
>            Assignee: Plamen Jeliazkov
>         Attachments: HADOOP-10641.patch, HADOOP-10641.patch, HADOOP-10641.patch, hadoop-coordination.patch
> Coordination Engine (CE) is a system, which allows to agree on a sequence of events in
a distributed system. In order to be reliable CE should be distributed by itself.
> Coordination Engine can be based on different algorithms (paxos, raft, 2PC, zab) and
have different implementations, depending on use cases, reliability, availability, and performance
> CE should have a common API, so that it could serve as a pluggable component in different
projects. The immediate beneficiaries are HDFS (HDFS-6469) and HBase (HBASE-10909).
> First implementation is proposed to be based on ZooKeeper.

This message was sent by Atlassian JIRA

View raw message