hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10614) Appended blocks can be closed even before IBRs from DataNodes
Date Fri, 15 Jul 2016 18:00:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379800#comment-15379800

Jing Zhao commented on HDFS-10614:

So is it possible that we have the following scenario: 
1. NN receives block_received msg from DN1 with GS 1001. Because the client has not sent the
commit block request yet, the block is not COMPLETE after this msg.
2. Somehow an old DN1's block_receiving msg with blk GS 1000 was sent to NN after #1 (I do
not think currently this can happen, but let's say a new bug in DN causes this)
3. NN finds that the block_receiving msg reports a replica which is not in FINALIZED state,
thus removes the block from the storage (and because the blockInfo's is not in COMPLETE state
thus {{checkReplicaCorrupt}} passes).

The above #2 may never happen, but currently my feeling is it can be safer to add the GS check
on the NN side. What do you think, [~vinayrpet]?

> Appended blocks can be closed even before IBRs from DataNodes
> -------------------------------------------------------------
>                 Key: HDFS-10614
>                 URL: https://issues.apache.org/jira/browse/HDFS-10614
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Vinayakumar B
>            Assignee: Vinayakumar B
>         Attachments: HDFS-10614.01.patch, HDFS-10614.02.patch
> Scenario:
>    1. Open the file for append()
>    2. Trigger append pipeline setup by adding some data.
>    3. Consider RECEIVING IBRs of DNs reaches NN first.
>    4. updatePipeline() rpc sent to namenode to update the pipeline.
>    5. Now, if complete() is called on the file even before closing the pipeline, then
block will be COMPLETE, even before block is actually FINALIZED at DN side and file will be

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message