hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Plamen Jeliazkov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10641) Introduce Coordination Engine
Date Fri, 06 Jun 2014 21:56:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14020426#comment-14020426
] 

Plamen Jeliazkov commented on HADOOP-10641:
-------------------------------------------

[~szetszwo], thanks for the review! It was nice to finally see a face at the Summit as well.
:)
[~atm], thanks for the comments! I think I am outside of that discussion, most likely [~cos]
or [~shv] can comment better on where to take the project.

I posted a new patch around the same time your review came in; there were mistakes in the
way agreement executions work.
* ProtoBuf is certainly a nice choice for serialization. However, we shouldn't need to bind
ourselves to any one serialization format. This is why we use Serializable. It is certainly
possible to have the writeObject call write out a ProtoBuf of the proposal itself, for example,
and read the values back using ProtoBuf as well. This is feasible with the current interfaces.
* Good point on version compatibility. AFAIK, version compatibility would take place once
the quorum is established as prior to that there is no communication between the engines.
So the coordination engine, as part of bootstrap, should perform a version check against its
quorum peers. Perhaps this means extending the API, or making it part of a larger interface?
(VersionedCoordinationEngine)? [~shv] might be able to comment better.
* Please see my new patch. The idea is indeed to make the agreement execute on some callBack
object, SampleLearner in this case. The new patch should show the test making use of it.
* Yes we can probably do some refactoring here. I'll work on a new patch.
* Yes we can add details for ZkCoordinationEngine. Unsure of any clear advantages and disadvantages.
The only thing that comes to my mind right away is that it may be possible to build Paxos
directly into the CoordinationEngine implementation, thus co-locating the coordination service
with the server / application itself, rather than having to make RPC calls and wait for responses,
like with ZooKeeper(s). I don't think the intent of this work is really to compare any one
coordination mechanism with another but so much as provide a common interface for which one
can implement whichever they prefer.

> Introduce Coordination Engine
> -----------------------------
>
>                 Key: HADOOP-10641
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10641
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 3.0.0
>            Reporter: Konstantin Shvachko
>            Assignee: Plamen Jeliazkov
>         Attachments: HADOOP-10641.patch, HADOOP-10641.patch, HADOOP-10641.patch
>
>
> Coordination Engine (CE) is a system, which allows to agree on a sequence of events in
a distributed system. In order to be reliable CE should be distributed by itself.
> Coordination Engine can be based on different algorithms (paxos, raft, 2PC, zab) and
have different implementations, depending on use cases, reliability, availability, and performance
requirements.
> CE should have a common API, so that it could serve as a pluggable component in different
projects. The immediate beneficiaries are HDFS (HDFS-6469) and HBase (HBASE-10909).
> First implementation is proposed to be based on ZooKeeper.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message