hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2932) Under replicated block after the pipeline recovery.
Date Mon, 15 Sep 2014 08:56:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133717#comment-14133717

Vinayakumar B commented on HDFS-2932:

Hi [~usrikanth],
Thanks for looking into this issue.

1. Case1  is a solved in HDFS-3493

2. Case 2: I understand your point,
    I agree that it will be better to detect the failed replicas as early as possible and
delete it. but we cannot do that on the fly while writing itself. It may delete the working
copy itself if the report from datanode is little delayed. and this is possible in case of
huge cluster. Its better to keep the corrupt replica for sometime instead of loosing the valid
replica. Similar cases observed and code has been added to ignore such variations. see below
comment in BlockManager.java.
{code}          // If it's a RBW report for a COMPLETE block, it may just be that
          // the block report got a little bit delayed after the pipeline
          // closed. So, ignore this report, assuming we will get a
          // FINALIZED replica later. See HDFS-2791{code}
Solution is to be able to detect and capture a write-pipeline-failed replica as early as possible.
First fix may be to change the check from 'isCompleted' to 'isCommitted'. This will capture
write-pipeline-failed replicas reported just after commit and before 'complete' and mark them
as corrupt.
 the timegap between 'isCommitted' and 'isCompleted' is not so huge, so ideally this will
not change much.

Then to capture write-pipeline-failed replicas reported before commit, I am investigating
if this can be solved by marking them as corrupt as part of commit. There already exists a
check to find any mis-stamped replicas during commit but we only remove them from the blocksMap.
In addition can we not mark such replicas as corrupt?
"setGenerationStampAndVerifyReplicas" is just updating the inmemory states of the replicas
being written in Namenode. its not changing the blocksMap. I dont think this is the right
place to decide about the corrupt replicas.

I think its always better to handle the block validations when reported from datanode, yes
it takes time :(

> Under replicated block after the pipeline recovery.
> ---------------------------------------------------
>                 Key: HDFS-2932
>                 URL: https://issues.apache.org/jira/browse/HDFS-2932
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 0.24.0
>            Reporter: J.Andreina
>             Fix For: 0.24.0
> Started 1NN,DN1,DN2,DN3 in the same machine.
> Written a huge file of size 2 Gb
> while the write for the block-id-1005 is in progress bruought down DN3.
> after the pipeline recovery happened.Block stamp changed into block_id_1006 in DN1,Dn2.
> after the write is over.DN3 is brought up and fsck command is issued.
> the following mess is displayed as follows
> "block-id_1006 is underreplicatede.Target replicas is 3 but found 2 replicas".

This message was sent by Atlassian JIRA

View raw message