hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8453) Erasure coding: properly assign start offset for internal blocks in a block group
Date Wed, 27 May 2015 01:19:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560258#comment-14560258
] 

Walter Su commented on HDFS-8453:
---------------------------------

I didn't see any difference between {{refreshLocatedBlock}} and old version's {{getBlockAt}}
This is old version
{code}
protected LocatedBlock getBlockAt(long blkStartOffset) 
{code}
This is new version
{code}
protected LocatedBlock refreshLocatedBlock(LocatedBlock block) {
  LocatedBlock lb = getBlockGroupAt(block.getStartOffset());
{code}
You didn't pass blkStartOffset, you get blkStartOffset from inside. So there is no difference.
This line really matters. It resolves the issue.
{code}
+          bg.getStartOffset(), bg.isCorrupt(), null);
{code}
The solution is to make it meaningless.

> Erasure coding: properly assign start offset for internal blocks in a block group
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-8453
>                 URL: https://issues.apache.org/jira/browse/HDFS-8453
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8453-HDFS-7285.00.patch
>
>
> {code}
>   void actualGetFromOneDataNode(final DNAddrPair datanode,
>     ...
>       LocatedBlock block = getBlockAt(blockStartOffset);
>     ...
>       fetchBlockAt(block.getStartOffset());
> {code}
> The {{blockStartOffset}} here is from inner block. For parity blocks, the offset will
overlap with the next block group, and we may end up with fetching wrong block. So we have
to assign a meaningful start offset for internal blocks in a block group, especially for parity
blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message