hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Powell (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5194) Robust support for alternate FsDatasetSpi implementations
Date Thu, 05 Dec 2013 00:39:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13839546#comment-13839546

David Powell commented on HDFS-5194:

As Eli chronicles above, we decided it made sense to break up the work
proposed by the attached document into smaller pieces that could be
evaluated individually.  The breakdown I have in mind are roughly along
the lines of the subsections of section 4, "Implementation details":

1) Expose any interfaces needed by a FsDatasetSpi subclass.
2) Reduce / eliminate java.io.File dependencies in FsDatasetSpi
   and support classes.
3) Address assumptions regarding FsDatasetSpi implementation in the
   common portions of the Datanode.
4) Introduce interfaces permitting an FsDatasetSpi implementation to
   provide short-circuit functionality.

The scope of #2 and #3 are be broad enough that multiple sub-tasks
are likely to emerge from each.

Though it is listed separately in the document, testing in general should
not be considered an independent effort and should instead accompany 
changes made for any of the above pieces.  That said, there may be a need
for some pure-testing activities, such as: changes to make it easier 
to apply existing HDFS tests to alternate dataset implementations, the
addition of tests to detect behavioural change in interfaces required
by an alternate dataset implementation, and other changes that don't
naturally align with pure implementation work.

> Robust support for alternate FsDatasetSpi implementations
> ---------------------------------------------------------
>                 Key: HDFS-5194
>                 URL: https://issues.apache.org/jira/browse/HDFS-5194
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, hdfs-client
>            Reporter: David Powell
>            Priority: Minor
>         Attachments: HDFS-5194.design.09112013.pdf, HDFS-5194.patch.09112013
> The existing FsDatasetSpi interface is well-positioned to permit extending Hadoop to
run natively on non-traditional storage architectures.  Before this can be done, however,
a number of gaps need to be addressed.  This JIRA documents those gaps, suggests some solutions,
and puts forth a sample implementation of some of the key changes needed.

This message was sent by Atlassian JIRA

View raw message