hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wellington Chevreuil (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12182) BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
Date Mon, 31 Jul 2017 12:43:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107262#comment-16107262
] 

Wellington Chevreuil commented on HDFS-12182:
---------------------------------------------

I believe the failed tests on last patch are not related, as the same are passing locally.
The findbugs warning shouldn't be related either, it's all referring to classes not touched
by the patch.

> BlockManager.metaSave does not distinguish between "under replicated" and "missing" blocks
> ------------------------------------------------------------------------------------------
>
>                 Key: HDFS-12182
>                 URL: https://issues.apache.org/jira/browse/HDFS-12182
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Wellington Chevreuil
>            Assignee: Wellington Chevreuil
>            Priority: Trivial
>              Labels: newbie
>             Fix For: 3.0.0-alpha3
>
>         Attachments: HDFS-12182.001.patch, HDFS-12182.002.patch, HDFS-12182.003.patch
>
>
> Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs CLI command)
reports both "under replicated" and "missing" blocks under same metric *Metasave: Blocks waiting
for reconstruction:* as shown on below code snippet:
> {noformat}
>    synchronized (neededReconstruction) {
>       out.println("Metasave: Blocks waiting for reconstruction: "
>           + neededReconstruction.size());
>       for (Block block : neededReconstruction) {
>         dumpBlockMeta(block, out);
>       }
>     }
> {noformat}
> *neededReconstruction* is an instance of *LowRedundancyBlocks*, which actually wraps
5 priority queues currently. 4 of these queues store different under replicated scenarios,
but the 5th one is dedicated for corrupt/missing blocks. 
> Thus, metasave report may suggest some corrupt blocks are just under replicated. This
can be misleading for admins and operators trying to track block missing/corruption issues,
and/or other issues related to *BlockManager* metrics.
> I would like to propose a patch with trivial changes that would report corrupt blocks
separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message