hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5366) recaching improvements
Date Wed, 13 Nov 2013 02:10:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820835#comment-13820835
] 

Colin Patrick McCabe commented on HDFS-5366:
--------------------------------------------

bq. The mlock stubbing will need to be rebased a bit for HDFS-5450 which was just committed,
since it was still stubbing with MappableBlock#Mlocker.

rebased in version 7

bq. There was also a fix to TestFsDatasetCache to reset the Mlocker stub to the default after
it's set, will want the same thing here.

yeah.

bq. We probably want to stub mlock for TestPathBasedCacheRequests too, but I feel like we
should leave at least one test calling mlock. Maybe in TestNativeIO?

Yeah, TestNativeIO still tests "raw" mlock.

> recaching improvements
> ----------------------
>
>                 Key: HDFS-5366
>                 URL: https://issues.apache.org/jira/browse/HDFS-5366
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5366-caching.001.patch, HDFS-5366.002.patch, HDFS-5366.005.patch,
HDFS-5366.007.patch
>
>
> There are a few things about our HDFS-4949 recaching strategy that could be improved.
> * We should monitor the DN's maximum and current mlock'ed memory consumption levels,
so that we don't ask the DN to do stuff it can't.
> * We should not try to initiate caching on stale or decomissioning DataNodes (although
we should not recache things stored on such nodes until they're declared dead).
> * We might want to resend the {{DNA_CACHE}} or {{DNA_UNCACHE}} command a few times before
giving up.  Currently, we only send it once.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message