hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-4949) Centralized cache management in HDFS
Date Fri, 09 Aug 2013 21:30:49 GMT

     [ https://issues.apache.org/jira/browse/HDFS-4949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Andrew Wang updated HDFS-4949:

    Attachment: caching-design-doc-2013-08-09.pdf

Suresh, thanks for posting your notes. Attached is a revised design doc that beefs up the
resource management / user quotas section, as well as addressing your other smaller points.

As a meta-point, I think much of the remaining resource management design can wait until after
we get the initial end-to-end implementation going. I think it's reasonable for the first
iteration to do something simple like "superuser only" or user quotas, then we layer on the
complexities of pools, priorities, ACLs, min/max/share, and failure cases afterwards. It's
good to get the API roughly right so we code with foresight, but I don't see us getting around
to implementing pools for at least a month or two.
> Centralized cache management in HDFS
> ------------------------------------
>                 Key: HDFS-4949
>                 URL: https://issues.apache.org/jira/browse/HDFS-4949
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, namenode
>    Affects Versions: 3.0.0, 2.3.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>         Attachments: caching-design-doc-2013-07-02.pdf, caching-design-doc-2013-08-09.pdf
> HDFS currently has no support for managing or exposing in-memory caches at datanodes.
This makes it harder for higher level application frameworks like Hive, Pig, and Impala to
effectively use cluster memory, because they cannot explicitly cache important datasets or
place their tasks for memory locality.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message