Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 29 Sep 2016 15:06:20 +0000 (UTC)
From: "Daryn Sharp (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13006761.1474526642000.698071.1475161580795@Atlassian.JIRA>
In-Reply-To: <JIRA.13006761.1474526642000@Atlassian.JIRA>
References: <JIRA.13006761.1474526642000@Atlassian.JIRA> <JIRA.13006761.1474526642716@arcas>
Subject: [jira] [Commented] (HDFS-10887) Provide admin/debug tool to dump
 block map
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 29 Sep 2016 15:06:23 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533020#comment-15533020 ] 

Daryn Sharp commented on HDFS-10887:
------------------------------------

Agree with Kihwal that a tool that  lists which nodes in the include list haven't block reported would be far more useful.  Technically, traversing a blocks map containing over well over 1/4 billion blocks isn't going to be cheap.  Ignoring that...

Operationally and in practice, the results of a block scan wouldn't be useful.  Finding the blocks with a distributed find (which I hope you aren't suggesting hdfs should support) presumes that nodes with "missing" blocks are online.  In most cases the problem is a hardware failure, ex. host or switch or failed volume.  You can't find what's not available.

Knowing which hosts are dead, not reporting, have failed storages lets you immediately know where to focus attention.  If the node is down, bring it up.  If it's up and hasn't reported, use dfsadmin to force a block report.

> Provide admin/debug tool to dump block map
> ------------------------------------------
>
>                 Key: HDFS-10887
>                 URL: https://issues.apache.org/jira/browse/HDFS-10887
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs, namenode
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-10887.001.patch, HDFS-10887.002.patch
>
>
> From time to time, when NN restarts, we see
> {code}
> "The reported blocks X needs additional Y blocks to reach the threshold 0.9990 of total blocks Z. Safe mode will be turned off automatically.
> {code}
> We'd wonder what these blocks that still need block reports are, and what DNs they could possibly be located, what happened to these DNs.
> This jira to to propose a new admin or debug tool to dump the block map info with the blocks that have fewer than minRepl replicas.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org