hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9289) check genStamp when complete file
Date Wed, 28 Oct 2015 22:21:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979367#comment-14979367

Zhe Zhang commented on HDFS-9289:

Thanks Jing for sharing the thoughts.

I think the GS validation in {{BlockManager#commitBlock}} is a little tricky. What if the
{{updatePipeline}} RPC call has successfully finished NN side changes but failed in sending
response to client? Should we allow the client to commit the block?

GS is used to determine whether a replica is stale, but the client doesn't have a replica.
Among the 3 attributes of a block (ID, size, GS), client should always have the same ID as
NN, and should always have fresher size than NN. So maybe the right thing to do is to discard
the client reported GS in {{commitBlock}}, but I'm not so sure.

How about we use this JIRA to commit the {{volatile}} change (which should fix the reported
issue) and dedicate a follow-on JIRA to the {{commitBlock}} GS validation change?

> check genStamp when complete file
> ---------------------------------
>                 Key: HDFS-9289
>                 URL: https://issues.apache.org/jira/browse/HDFS-9289
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Chang Li
>            Assignee: Chang Li
>            Priority: Critical
>         Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, HDFS-9289.4.patch
> we have seen a case of corrupt block which is caused by file complete after a pipelineUpdate,
but the file complete with the old block genStamp. This caused the replicas of two datanodes
in updated pipeline to be viewed as corrupte. Propose to check genstamp when commit block

This message was sent by Atlassian JIRA

View raw message