hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
Date Tue, 10 Jun 2014 01:58:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026025#comment-14026025

Aaron T. Myers commented on HADOOP-10641:

bq. Not sure where this leaves QJM. I thought it satisfies all these requirements including
the Hadoop's aim.

The NameNode QuorumJournalManager and JNs are expressly for storing HDFS NN edit logs. Not
for general purpose consensus, not for use by other projects like HBase, etc.

bq. Hadoop common is chosen for the coordination engine interface as the lowest common ancestor.

But a separate project could just as well be a common ancestor, just like both Hadoop and
HBase separately depend on ZooKeeper. There's no actual need for it to be in Hadoop Common
if HBase is to use it.

bq. It could be used for anything Hadoop from here: HDFS, HBase, Yarn. 

But seems like it could also be used for arbitrary, non-Hadoop things, correct? If so, why
put it in Hadoop?

bq. I don't have other use cases for it in my mind at the moment. Did not have enough experience
with TLPs and Incubators, but thought it needs like something bigger. Say, in this case, implementations
for more than one CE. Sorry if I misunderstood you, but if you want to take it to Apache Incubator
I'll be on your side.

I personally don't think there's any good reason for this to start out as part of a larger
project, and honestly think there are several downsides. For example, Hadoop's release cadence
is too slow for a new project like this, there's not much expertise in Hadoop for the general
problem of distributed consensus, possible desire for other non-Hadoop projects to want to
use it, etc.

> Introduce Coordination Engine
> -----------------------------
>                 Key: HADOOP-10641
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10641
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 3.0.0
>            Reporter: Konstantin Shvachko
>            Assignee: Plamen Jeliazkov
>         Attachments: HADOOP-10641.patch, HADOOP-10641.patch, HADOOP-10641.patch
> Coordination Engine (CE) is a system, which allows to agree on a sequence of events in
a distributed system. In order to be reliable CE should be distributed by itself.
> Coordination Engine can be based on different algorithms (paxos, raft, 2PC, zab) and
have different implementations, depending on use cases, reliability, availability, and performance
> CE should have a common API, so that it could serve as a pluggable component in different
projects. The immediate beneficiaries are HDFS (HDFS-6469) and HBase (HBASE-10909).
> First implementation is proposed to be based on ZooKeeper.

This message was sent by Atlassian JIRA

View raw message