hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8580) Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
Date Wed, 30 Sep 2015 19:35:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938386#comment-14938386
] 

Hudson commented on HDFS-8580:
------------------------------

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/])
HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and (jing9: rev c12a974ccf5f52f63e4f825d8b4d2385953cd119)
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStripedINodeFile.java
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfoStriped.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java
* hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
* hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStripedUnderConstruction.java


> Erasure coding: Persist cellSize in BlockInfoStriped and StripedBlocksFeature
> -----------------------------------------------------------------------------
>
>                 Key: HDFS-8580
>                 URL: https://issues.apache.org/jira/browse/HDFS-8580
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Walter Su
>             Fix For: HDFS-7285
>
>         Attachments: HDFS-8580-HDFS-7285.01.patch, HDFS-8580-HDFS-7285.02.patch, HDFS-8580-HDFS-7285.03.patch,
HDFS-8580-HDFS-7285.04.patch, HDFS-8580.00.patch
>
>
> Zhe Zhang, Kai Zheng and I had a offline discussion. Here is what we thought:  Add a
cellSize field in BlockInfoStriped as a workaround, and deal with memory usage in follow-on.(HDFS-8059)
> discussion in HDFS-8494:
> from Walter Su:
> {quote}
> I think BlockInfoStriped needs to keep cellSize.
> {quote}
> from [~vinayrpet]:
> {quote}
> I too was thinking the same when the FSImageLoader problem has came up. This will increase
the memory usage by ~4bytes for each block though.
> {quote}
> from [~jingzhao]
> {quote}
> -Also, we should consider adding a chunk size field to StripedBlockProto and removing
the cell size field from HdfsFileStatus. In this way we can access the chunk size information
in the storage layer.-
> {quote}
> ==
> update:
> from [~jingzhao]
> {quote}
> For fsimage part, since HDFS-8585 just removes StripedBlockProto, I guess what we can
do here is to either 1) add the cellSize information into StripedBlocksFeature in fsimage.proto,
or 2) bring StripedBlockProto back and put block info and cell size there.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message