hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8401) Memfs - a layered file system for in-memory storage in HDFS
Date Thu, 28 May 2015 19:56:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14563580#comment-14563580

Andrew Wang commented on HDFS-8401:

bq. It's impractical to involve the administrator every time a new file is to be cached.

Read caching can be done by normal users, not just admins. We also have directory-level cache
directives which kick in automatically without any explicit user involvement

If some of the other enhancements you mention could be built into HDFS, that'd also be preferable
(de-dupe, predictability (?)).

Anecdotal, but I've heard a lot of users say that changing the scheme is not an option for
them. If your concern is ease of use, focusing on improvements to what we already have in
HDFS might be more bang for the buck. We have the LAZY_PERSIST storage policy and directory-level
cache directives which seem like a start. Colin also mentioned opportunistic cache directives,
which would be a really nice enhancement.

> Memfs - a layered file system for in-memory storage in HDFS
> -----------------------------------------------------------
>                 Key: HDFS-8401
>                 URL: https://issues.apache.org/jira/browse/HDFS-8401
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
> We propose creating a layered filesystem that can provide in-memory storage using existing
features within HDFS. memfs will use lazy persist writes introduced by HDFS-6581. For reads,
memfs can use the Centralized Cache Management feature introduced in HDFS-4949 to load hot
data to memory.
> Paths in memfs and hdfs will correspond 1:1 so memfs will require no additional metadata
and it can be implemented entirely as a client-side library.
> The advantage of a layered file system is that it requires little or no changes to existing
applications. e.g. Applications can use something like {{memfs://}} instead of {{hdfs://}}
for files targeted to memory storage. 

This message was sent by Atlassian JIRA

View raw message