hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6919) Enforce a single limit for RAM disk usage and replicas cached via locking
Date Fri, 26 Sep 2014 01:47:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148608#comment-14148608
] 

Arpit Agarwal commented on HDFS-6919:
-------------------------------------

Colin, thinking ahead to this integration. Could you please describe what you have in mind?

I think as a first step it would be good to have the caching logic on the DN know the current
RAM disk usage, as well as the RAM disk replica management logic know how much memory is in
use by caching. So if one feature is not in use the other gets access to the entire memory.
We'd at least avoid any memory fragmentation issues. The total limit would be the {{dfs.datanode.max.locked.memory}}
or whatever we call it.

Longer term perhaps we can look at RAM disk manager evicting cold data from the DN cache to
make room for writes, and vice versa.

> Enforce a single limit for RAM disk usage and replicas cached via locking
> -------------------------------------------------------------------------
>
>                 Key: HDFS-6919
>                 URL: https://issues.apache.org/jira/browse/HDFS-6919
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Arpit Agarwal
>            Assignee: Colin Patrick McCabe
>            Priority: Blocker
>
> The DataNode can have a single limit for memory usage which applies to both replicas
cached via CCM and replicas on RAM disk.
> See comments [1|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106025&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106025],
[2|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106245&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106245]
and [3|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106575&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106575]
for discussion.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message