hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8058) Erasure coding: use BlockInfo[] for both striped and contiguous blocks in INodeFile
Date Wed, 15 Jul 2015 07:59:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Zhe Zhang updated HDFS-8058:
    Attachment: HDFS-8058-HDFS-7285.010.patch

Thanks Jing for noticing the {{TestDFSStripedOutputStreamWithFailure}} timeout.

It turns out to be a tricky bug. It happened between 06 and 07 patch. Basically I forgot to
carry over the {{setFileReplication((short) 0)}} logic in the new {{INodeFile}} constructor.
New patch addresses this issue:
      // Replication factor for striped files is zero
      if (isStriped) {
        h = REPLICATION.BITS.combine(0L, h);
        h = IS_STRIPED.BITS.combine(1L, h);
      } else {
        h = REPLICATION.BITS.combine(replication, h);
        h = IS_STRIPED.BITS.combine(0L, h);

> Erasure coding: use BlockInfo[] for both striped and contiguous blocks in INodeFile
> -----------------------------------------------------------------------------------
>                 Key: HDFS-8058
>                 URL: https://issues.apache.org/jira/browse/HDFS-8058
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: HDFS-7285
>            Reporter: Yi Liu
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8058-HDFS-7285.003.patch, HDFS-8058-HDFS-7285.004.patch, HDFS-8058-HDFS-7285.005.patch,
HDFS-8058-HDFS-7285.006.patch, HDFS-8058-HDFS-7285.007.patch, HDFS-8058-HDFS-7285.008.patch,
HDFS-8058-HDFS-7285.009.patch, HDFS-8058-HDFS-7285.010.patch, HDFS-8058.001.patch, HDFS-8058.002.patch
> This JIRA is to use {{BlockInfo[] blocks}} for both striped and contiguous blocks in
> Currently {{FileWithStripedBlocksFeature}} keeps separate list for striped blocks, and
the methods there duplicate with those in INodeFile, and current code need to judge {{isStriped}}
then do different things. Also if file is striped, the {{blocks}} in INodeFile occupy a reference
memory space.
> These are not necessary, and we can use the same {{blocks}} to make code more clear.
> I keep {{FileWithStripedBlocksFeature}} as empty for follow use: I will file a new JIRA
to move {{dataBlockNum}} and {{parityBlockNum}} from *BlockInfoStriped* to INodeFile, since
ideally they are the same for all striped blocks in a file, and store them in block will waste
NN memory.

This message was sent by Atlassian JIRA

View raw message