hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10887) Provide admin/debug tool to dump block map
Date Thu, 29 Sep 2016 21:55:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15534176#comment-15534176
] 

Yongjun Zhang commented on HDFS-10887:
--------------------------------------

Thanks a lot [~daryn]. 

So we can have an additional structure such as a set of DNs in NN to record the DNs that have
not reported. The set is initialized with the include list, and when NN receives a block report
from the DN, it will remove DN from the set; when NN claims a DN to be dead (after 10.5 minutes
no heartbeating), it will add the DN to the set. Then we can have an api to report this set.
This will incur some additional cost on processing each block report (check whether the DN
exists in the set, remove if it does)

The above solution is doable when include list used, we need a way to handle the situation
when include list is not used. I guess it's rare that include list is not used though. Do
you guys agree to have the above mentioned solution and to ignore the case that include list
is not used? [~daryn] [~kihwal]?

Thanks.



> Provide admin/debug tool to dump block map
> ------------------------------------------
>
>                 Key: HDFS-10887
>                 URL: https://issues.apache.org/jira/browse/HDFS-10887
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs, namenode
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-10887.001.patch, HDFS-10887.002.patch
>
>
> From time to time, when NN restarts, we see
> {code}
> "The reported blocks X needs additional Y blocks to reach the threshold 0.9990 of total
blocks Z. Safe mode will be turned off automatically.
> {code}
> We'd wonder what these blocks that still need block reports are, and what DNs they could
possibly be located, what happened to these DNs.
> This jira to to propose a new admin or debug tool to dump the block map info with the
blocks that have fewer than minRepl replicas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message