hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5324) Make Namespace implementation pluggable in the namenode
Date Wed, 09 Oct 2013 17:47:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790652#comment-13790652
] 

Sanjay Radia commented on HDFS-5324:
------------------------------------

I am copying my comment from  hdfs-dev:

HDFS pluggability (and relation to pluggability added as part of Federation)
 * Pluggabilty and federation are orthogonal, although we did improved the pluggabily of HDFS
as part of federation implementation. The *block layer* was separated out as part of the federation
work and hence makes the general development of new  of HDFS namespace implementations easier.
 Federation's  pluggablity was  targeted towards  someone writing a new NN and reusing the
block storage layer via a library   and *optionally* living side-by-side with different implementations
of the NN *within the same cluster*. Hence we added notion of block pools and separated out
the block management layer.  
* So your proposed work is clearly not in conflict with Federation or even with the pluggability
that Federation added, but philosophically,  your proposal is complementary. 

Considerations: A Public API?
The FileSystem/AbstractFileSystem APIs and the newly proposed AbstractFSNamesystem are targeting
very different kinds of plugability into Hadoop. The former takes a thin application API (FileSystem
and FileContext) and makes it easy for users to plug in different filesytems (S3, LocalFS,
etc) as Hadoop compatible filesystems. In contrast the later (the proposed AbstractFSNamesystem)
is a fatter interface inside the depths of HDFS implementation and makes parts of the impl
pluggable. 

I would  not make your proposed AbstractFSNamesystem a public stable Hadoop API but instead
direct it towards to HDFS developers who want to extend the implementation of HDFS more easily.
Were you envisioning the AbstractFSNamesystem to be a stable public Hadoop API? If someone
has their own private implementation for this new abstract class, would  the HDFS community
have the freedom to modify the abstract class in incompatible ways? 

> Make Namespace implementation pluggable in the namenode
> -------------------------------------------------------
>
>                 Key: HDFS-5324
>                 URL: https://issues.apache.org/jira/browse/HDFS-5324
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.1.1-beta
>         Environment: All
>            Reporter: Milind Bhandarkar
>            Assignee: Milind Bhandarkar
>             Fix For: 3.0.0
>
>
> For the last couple of months, we have been working on making Namespace
> implementation in the namenode pluggable. We have demonstrated that it can
> be done without major surgery on the namenode, and does not have noticeable
> performance impact. We would like to contribute it back to Apache HDFS.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message