hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7853) Erasure coding: extend LocatedBlocks to support reading from striped files
Date Thu, 05 Mar 2015 00:34:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347836#comment-14347836
] 

Zhe Zhang commented on HDFS-7853:
---------------------------------

The extended {{DFSStripedInputStream}} uses {{LocatedStripedBlock}} like the following:
{code}
 private LocatedBlock[] parseStripedBlockGroup(LocatedBlock bg) {
    LocatedBlock[] lbs = new LocatedBlock[HdfsConstants.NUM_DATA_BLOCKS];
    for (short i = 0; i < HdfsConstants.NUM_DATA_BLOCKS; i++) {
      ExtendedBlock blk = new ExtendedBlock(bg.getBlock());
      short j = bg instanceof LocatedStripedBlock ?
          ((LocatedStripedBlock) bg).getBlockIndicies()[i] : i;
      blk.setBlockId(bg.getBlock().getBlockId() + i);

      lbs[j] = new LocatedBlock(blk, new DatanodeInfo[]{bg.getLocations()[i]},
          new String[]{bg.getStorageIDs()[i]},
          new StorageType[]{bg.getStorageTypes()[i]},
          bg.getStartOffset() + i * cellSize, bg.isCorrupt(), null);
    }
    return lbs;
  }
{code}

My [test|https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14347808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14347808]
writes a file with 2 blocks and reads it back. I found {{blockIndices}} is always sequential
({{getBlockIndicies()\[i\]}} always equal to i ). But the stored locations are not necessarily
sorted based on indices in the group. So sometimes the test passes and sometimes it gets ReplicaNotFound
error.

My test manually applied HDFS-7729 though. Without it, I guess we should add synthetic block
reports to unit-test the above point?

> Erasure coding: extend LocatedBlocks to support reading from striped files
> --------------------------------------------------------------------------
>
>                 Key: HDFS-7853
>                 URL: https://issues.apache.org/jira/browse/HDFS-7853
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Jing Zhao
>         Attachments: HDFS-7853.000.patch
>
>
> We should extend {{LocatedBlocks}} class so {{getBlockLocations}} can work with striping
layout (possibly an extra list specifying the index of each location in the group)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message