hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5366) recaching improvements
Date Tue, 12 Nov 2013 19:06:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820361#comment-13820361

Colin Patrick McCabe commented on HDFS-5366:

bq. Since we already have a config key named "dfs.namenode.path.based.cache.refresh.interval.ms",
can we call this one "dfs.namenode.path.based.cache.retry.interval.ms"?

yeah, I like that name better

bq. New configs should go in hdfs-default.xml too


bq. Nit: extra newline in DatanodeManager#getCacheCommand


bq. Javadoc on DatanodeDescriptor methods saying whether they take wallclock or monotonic


bq. Any reason to prefer the iterator-based removal over using clear? If it's not necessary,
we could not do this to keep the diff small.

This way, we only have to iterate over it once, not twice.

bq. Extra imports in CacheReplicationMonitor, DatanodeDescriptor


bq. In DatanodeManager, having variables named sendingCachingCommands and sendCachingCommands
is confusing, rename to retryCachingCommands or something?

renamed to {{shouldSendCachingCommands}}

> recaching improvements
> ----------------------
>                 Key: HDFS-5366
>                 URL: https://issues.apache.org/jira/browse/HDFS-5366
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5366-caching.001.patch, HDFS-5366.002.patch
> There are a few things about our HDFS-4949 recaching strategy that could be improved.
> * We should monitor the DN's maximum and current mlock'ed memory consumption levels,
so that we don't ask the DN to do stuff it can't.
> * We should not try to initiate caching on stale or decomissioning DataNodes (although
we should not recache things stored on such nodes until they're declared dead).
> * We might want to resend the {{DNA_CACHE}} or {{DNA_UNCACHE}} command a few times before
giving up.  Currently, we only send it once.

This message was sent by Atlassian JIRA

View raw message