hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8030) HDFS Erasure Coding Phase II -- EC with contiguous layout
Date Tue, 15 Mar 2016 00:07:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15194425#comment-15194425

Uma Maheswara Rao G commented on HDFS-8030:

[~zhz] Me and Rakesh discussed again for some time on the document. Please find the points
to discuss/address for your opinion.
# As this design tries to convert files into EC mode from normal file layout, Blockgroups
needs to be created later when converting. But block groups generally we allocate continuous
blockids, but here how do we make that continuous blockids when converting?
# Does this create overheads on memory as we need to track blockGroups separately and if the
blockids are not continuous as discussed in #1
#  parity creation by reading whole 6blocks which will be 6*256MB into memory. Do we need
to think more on this point I think. We may need to think like contiguous blockid but Parity
generation based on Stripes?
Blk_0, Blk_1…Blk_5 are the contiguous blocks. We read cell from each block and just treat
as stripe and generate 3 parity. Continue until finishes all all data in blocks. Need to think
more on this.
# Do we support mixed zone? contains both striped files and contiguous EC files?
Please others also review the document and get the feedbacks.

> HDFS Erasure Coding Phase II -- EC with contiguous layout
> ---------------------------------------------------------
>                 Key: HDFS-8030
>                 URL: https://issues.apache.org/jira/browse/HDFS-8030
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: erasure-coding
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFSErasureCodingPhaseII-20151204.pdf
> Data redundancy form -- replication or erasure coding, should be orthogonal to block
layout -- contiguous or striped. This JIRA explores the combination of {{Erasure Coding}}
+ {{Contiguous}} block layout.
> As will be detailed in the design document, key benefits include preserving block locality,
and easy conversion between hot and cold modes. 

This message was sent by Atlassian JIRA

View raw message