hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7007) Interfaces to plugin ConsensusNode.
Date Tue, 10 Mar 2015 06:59:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354427#comment-14354427

Konstantin Shvachko commented on HDFS-7007:

h3. Possible approaches
# My original idea was to subclass certain NameNode classes to enable sending proposals for
the write RPC calls. The advantages of this approach include
#- that the entire ConsensusNode implementation can be kept completely separate as a different
subproject, like hadoop-hdfs-nfs or hadoop-kms,
#- and allow code reuse from the NameNode
#- While the argument against is too tight integration with NN, as per [~tlipcon] and [~atm]
# [~stevel] proposed to make NameNode use {{CompositeService}}. I see how that helps adding
components to the NameNode and maintaining their starts and stops, but I don't see how it
helps enabling coordination or avoiding subclassing.
# [~szetszwo] proposed to introduce an interface for {{FSNamesystem}} similar to DN's {{FsDatasetSpi}},
as in HDFS-5194. I do not have a proposal for that, but if this is what people want, I can
think of one. I see one potential disadvantage of this approach, which was mentioned for {{FsDatasetSpi}}
in HDFS-5751, that there is code duplication in implementations.
In the end CNode must create a directory same as NN. It would be good to have them call the
same mkdirs() method rather than duplicating it in both.
# Somebody, probably [~sanjay.radia], mentioned "in-line" approach. Akin to Standby NN, where
HAState distinguishes active from standby with execution depending on respective if statements.
# A rather low-level approach is to invoke coordination in the {{RpcEngine}} layer. That is,
when RPC call arrives we either (writes) submit a proposal to CE or (reads) call {{NameNodeRpcServer}}
directly as in regular NN. We can annotate NN protocol methods as writes or reads and RpcEngine
will invoke coordination depending on the annotation.
The advantage here is that it allows submitting proposals without fully deserializing the
RPC call. There will be still changes to FSNamesystem, since some write calls need or can
be optimized with extra checks before proposing, like safeMode and permissions for mkdirs(),
or choosing target locations before {{addBlock()}}.

We will probably end up using a combination of approaches. My favourite as of today is the
last one. It seems to be minimally invasive.

> Interfaces to plugin ConsensusNode.
> -----------------------------------
>                 Key: HDFS-7007
>                 URL: https://issues.apache.org/jira/browse/HDFS-7007
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Konstantin Shvachko
> This is to introduce interfaces in NameNode and namesystem, which are needed to plugin

This message was sent by Atlassian JIRA

View raw message