hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8376) Erasure Coding: Update last cellsize calculation according to whether the erasure codec has chunk boundary
Date Tue, 12 May 2015 12:16:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yi Liu updated HDFS-8376:
-------------------------
    Description: 
Current calculation for last cell size is as following. For parity cell, the last cell size
is the same as the first data cell.  But some erasure codec has chunk boundary, then the last
cellsize for parity block is the codec chunk size.
{code}
private static int lastCellSize(int size, int cellSize, int numDataBlocks,
      int i) {
    if (i < numDataBlocks) {
      // parity block size (i.e. i >= numDataBlocks) is the same as 
      // the first data block size (i.e. i = 0).
      size -= i*cellSize;
      if (size < 0) {
        size = 0;
      }
    }
    return size > cellSize? cellSize: size;
  }
{code}

  was:
Current calculation for last cell size is as following. For parity cell, the last cell size
is the same as the first data cell.  But as discussed in HDFS-8347 and on-line meeting, some
erasure codec is not linear, then the last cellsize for parity block is the codec chunk size.
{code}
private static int lastCellSize(int size, int cellSize, int numDataBlocks,
      int i) {
    if (i < numDataBlocks) {
      // parity block size (i.e. i >= numDataBlocks) is the same as 
      // the first data block size (i.e. i = 0).
      size -= i*cellSize;
      if (size < 0) {
        size = 0;
      }
    }
    return size > cellSize? cellSize: size;
  }
{code}

This JIRA also removes some duplicate code
{{StripedBlockUtil#constructStripedBlock}} and {{StripedBlockUtil#getStripedBlockLength}}
are duplicated with some other methods, we can clean up.

        Summary: Erasure Coding: Update last cellsize calculation according to whether the
erasure codec has chunk boundary  (was: Erasure Coding: Update last cellsize calculation according
to whether the erasure codec is linear)

> Erasure Coding: Update last cellsize calculation according to whether the erasure codec
has chunk boundary
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8376
>                 URL: https://issues.apache.org/jira/browse/HDFS-8376
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>
> Current calculation for last cell size is as following. For parity cell, the last cell
size is the same as the first data cell.  But some erasure codec has chunk boundary, then
the last cellsize for parity block is the codec chunk size.
> {code}
> private static int lastCellSize(int size, int cellSize, int numDataBlocks,
>       int i) {
>     if (i < numDataBlocks) {
>       // parity block size (i.e. i >= numDataBlocks) is the same as 
>       // the first data block size (i.e. i = 0).
>       size -= i*cellSize;
>       if (size < 0) {
>         size = 0;
>       }
>     }
>     return size > cellSize? cellSize: size;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message