hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8401) Memfs - a layered file system for in-memory storage in HDFS
Date Tue, 26 May 2015 22:37:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560011#comment-14560011

Arpit Agarwal commented on HDFS-8401:

bq. I'm not sure I see the advantage of having a separate file system, rather than simply
putting this into HDFS. bq. Can you clarify how users would interact with this system?
Colin, our goal is making it easier for applications to use memory support in HDFS specifically
and Hadoop Compatible File Systems in general.
# Allow using memory features without calling HDFS-specific APIs. This also isolates applications
from evolving APIs. Applications currently use shims and reflection tricks to work with different
versions of HDFS. 
# Once applications start using memfs someone could write a memfs layer over another HCFS
e.g. Amazon S3. 

memfs itself will not cache any data when used with hdfs. wrt interaction, applications can
choose to use {{memfs://}} paths instead of {{hdfs://}} paths for data targeted to memory.

bq. How does this relate to DDM?
There is no immediate plan to introduce a discardable namespace.

> Memfs - a layered file system for in-memory storage in HDFS
> -----------------------------------------------------------
>                 Key: HDFS-8401
>                 URL: https://issues.apache.org/jira/browse/HDFS-8401
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
> We propose creating a layered filesystem that can provide in-memory storage using existing
features within HDFS. memfs will use lazy persist writes introduced by HDFS-6581. For reads,
memfs can use the Centralized Cache Management feature introduced in HDFS-4949 to load hot
data to memory.
> Paths in memfs and hdfs will correspond 1:1 so memfs will require no additional metadata
and it can be implemented entirely as a client-side library.
> The advantage of a layered file system is that it requires little or no changes to existing
applications. e.g. Applications can use something like {{memfs://}} instead of {{hdfs://}}
for files targeted to memory storage. 

This message was sent by Atlassian JIRA

View raw message