hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5851) Support memory as a storage medium
Date Fri, 18 Apr 2014 21:04:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13974472#comment-13974472

Arpit Agarwal commented on HDFS-5851:

Hi Andrew,

bq. There are also plans to move towards sub-block caching. Whole-block caching is wasteful
for columnar formats like ORC and Parquet. With sub-block caching, automatic cache replacement
looks a lot more attractive (another planned feature). These are both things we can support
with HDFS-4949's infrastructure. I'm not sure about with HSM.
Does CCM support block-level caching or is it just _cache all blocks in a file_?

bq. Anyway, either pulling this towards HDFS-4949 or vice versa, we should figure out these
details before moving ahead. I'll echo Colin's desire for a meeting to discuss this. We're
willing to host at our Palo Alto office.
Thanks for the offer! We are discussing use cases and proposed API at high level. I am not
sure there is much overlap between CCM and our work and I expect them to solve different use
cases. However I am also open to discussing how and whether we can align them more closely
once we have shared our initial proposal.

> Support memory as a storage medium
> ----------------------------------
>                 Key: HDFS-5851
>                 URL: https://issues.apache.org/jira/browse/HDFS-5851
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
> Memory can be used as a storage medium for smaller/transient files for fast write throughput.
> More information/design will be added later.

This message was sent by Atlassian JIRA

View raw message