hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6093) Expose more caching information for debugging by users
Date Wed, 19 Mar 2014 22:51:43 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941108#comment-13941108

Andrew Wang commented on HDFS-6093:

Hi Arpit and Colin, thanks for reviewing. New patch is up. Addressed your feedback except
the following, and I also fixed a logging issue I found:

bq. update CentralizedCacheManagement.html in the docs?

Added a short blurb. A nice follow-on JIRA would be an FAQ for debugging caching, since it
can be tricky right now.

bq. display the pending caching/uncaching counts in the output of 'dfsadmin -report'?

I think dfsadmin -report is more about usage statistics than replication work. Having the
pending stats as a metric and on the webUI means it should still be easy enough to access.

bq. Was stillPendingUncached introduced to fix a bug?

This is required because cache reports just tell you what's cached, not also what was uncached.
So, we need to compute a diff to update pendingUncached correctly.

bq. <ternary statement code nit>

I prefer not to use ternary statements, so I'd like to leave it as is if that's okay.

bq. decouple the counter(s) that can be read from the CRM from the counters that the CRM uses

With the locking issues resolved, is it okay to just leave it with a single set of variables?
I could switch it over to AtomicLongs or something, but I think it's all under the FSN lock

bq. <colin> Incrementally updating the pendingUncached list and stats is a nice idea,
but it seems too ambitious for 2.4 at this point. 

I'm okay bumping this to 2.5 if you'd rather not put this in 2.4, but I think this all works
now with the locking fixed.

> Expose more caching information for debugging by users
> ------------------------------------------------------
>                 Key: HDFS-6093
>                 URL: https://issues.apache.org/jira/browse/HDFS-6093
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: caching
>    Affects Versions: 2.4.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>         Attachments: hdfs-6093-1.patch, hdfs-6093-2.patch
> When users submit a new cache directive, it's unclear if the NN has recognized it and
is actively trying to cache it, or if it's hung for some other reason. It'd be nice to expose
a "pending caching/uncaching" count the same way we expose pending replication work.
> It'd also be nice to display the aggregate cache capacity and usage in dfsadmin -report,
since we already have have it as a metric and expose it per-DN in report output.

This message was sent by Atlassian JIRA

View raw message