hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6093) Expose more caching information for debugging by users
Date Fri, 14 Mar 2014 22:46:44 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935758#comment-13935758
] 

Arpit Agarwal commented on HDFS-6093:
-------------------------------------

Hi Andrew,

I just tried this out your patch and I think there is some mismatch between the output of
{{dfsAdmin -report}} and {{cacheadmin -listPools}}.

This is with a single NN/single DN pseudocluster on Centos 6.5.

I ran the following commands:
- bin/hdfs cacheadmin -addPool pool1 -limit 1073741824
- bin/hdfs cacheadmin -addDirective -path /f1 -pool pool1

This says FILES_CACHED is zero.
{code}
$ bin/hdfs cacheadmin -listPools -stats
Found 1 result.
NAME   OWNER     GROUP     MODE             LIMIT  MAXTTL  BYTES_NEEDED  BYTES_CACHED  BYTES_OVERLIMIT
 FILES_NEEDED  FILES_CACHED
pool1  aagarwal  aagarwal  rwxr-xr-x   1073741824   never       1048576             0    
           0             1             0
{code}

However this says "cache used" is 1MB. 
{code}
aagarwal@arrow ~/deploy2/hadoop-3.0.0-SNAPSHOT$ bin/hdfs dfsadmin -report
Configured Capacity: 49202208768 (45.82 GB)
Present Capacity: 39676268544 (36.95 GB)
DFS Remaining: 39675179008 (36.95 GB)
DFS Used: 1089536 (1.04 MB)
DFS Used%: 0.00%

Configured Cache Capacity: 268435456 (256 MB)
Present Cache Capacity: 268435456 (256 MB)
Cache Remaining: 267386880 (255 MB)
Cache Used: 1048576 (1 MB)
Cache Used%: 0.39%
{code}

I did not see any error messages related to caching in the DN/NN logs.

> Expose more caching information for debugging by users
> ------------------------------------------------------
>
>                 Key: HDFS-6093
>                 URL: https://issues.apache.org/jira/browse/HDFS-6093
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: caching
>    Affects Versions: 2.4.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>         Attachments: hdfs-6093-1.patch
>
>
> When users submit a new cache directive, it's unclear if the NN has recognized it and
is actively trying to cache it, or if it's hung for some other reason. It'd be nice to expose
a "pending caching/uncaching" count the same way we expose pending replication work.
> It'd also be nice to display the aggregate cache capacity and usage in dfsadmin -report,
since we already have have it as a metric and expose it per-DN in report output.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message