hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5366) recaching improvements
Date Tue, 12 Nov 2013 19:06:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820361#comment-13820361
] 

Colin Patrick McCabe commented on HDFS-5366:
--------------------------------------------

bq. Since we already have a config key named "dfs.namenode.path.based.cache.refresh.interval.ms",
can we call this one "dfs.namenode.path.based.cache.retry.interval.ms"?

yeah, I like that name better

bq. New configs should go in hdfs-default.xml too

ok

bq. Nit: extra newline in DatanodeManager#getCacheCommand

ok

bq. Javadoc on DatanodeDescriptor methods saying whether they take wallclock or monotonic
time

ok

bq. Any reason to prefer the iterator-based removal over using clear? If it's not necessary,
we could not do this to keep the diff small.

This way, we only have to iterate over it once, not twice.

bq. Extra imports in CacheReplicationMonitor, DatanodeDescriptor

ok

bq. In DatanodeManager, having variables named sendingCachingCommands and sendCachingCommands
is confusing, rename to retryCachingCommands or something?

renamed to {{shouldSendCachingCommands}}

> recaching improvements
> ----------------------
>
>                 Key: HDFS-5366
>                 URL: https://issues.apache.org/jira/browse/HDFS-5366
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5366-caching.001.patch, HDFS-5366.002.patch
>
>
> There are a few things about our HDFS-4949 recaching strategy that could be improved.
> * We should monitor the DN's maximum and current mlock'ed memory consumption levels,
so that we don't ask the DN to do stuff it can't.
> * We should not try to initiate caching on stale or decomissioning DataNodes (although
we should not recache things stored on such nodes until they're declared dead).
> * We might want to resend the {{DNA_CACHE}} or {{DNA_UNCACHE}} command a few times before
giving up.  Currently, we only send it once.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message