hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8323) Bump GenerationStamp for write faliure in DFSStripedOutputStream
Date Mon, 18 May 2015 21:28:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549260#comment-14549260

Jing Zhao commented on HDFS-8323:

Thanks for updating the patch, Nicholas! Checking the patch again, and have a new question:
When bumping the GS, the LocatedStripedBlock contains the block indices directly retrieved
from the current UC blockInfo. Later on the client side, this indices determines how we parse
the LocatedStripedBlock into LocatedBlock[] and we expect this array contains (6 + 3) elements.
However, if the NN restarts while a long-running client writes the data, the indices returned
from NN depends on the block reports received by the NN. Thus it is possible the LocatedBlock
array parsed from LocatedStripedBlock contains <9 blocks.

The above scenario should be rare. However, it may be better to remove the dependency of the
block parsing part from the GS bumping logic. The information we need from the updateBlockForPipeline
RPC includes only the new GS number and the block token. In that sense, looks like we do not
need to parse the LocatedStripedBlock again in {{StripedDataStreamer#updateBlockForPipeline}},
but only need to refresh the GS/token?

> Bump GenerationStamp for write faliure in DFSStripedOutputStream
> ----------------------------------------------------------------
>                 Key: HDFS-8323
>                 URL: https://issues.apache.org/jira/browse/HDFS-8323
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h8323_20150511.patch, h8323_20150511b.patch, h8323_20150512.patch,

This message was sent by Atlassian JIRA

View raw message