hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8728) Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
Date Wed, 08 Jul 2015 17:52:05 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zhe Zhang updated HDFS-8728:
----------------------------
    Attachment: Merge-8-inodeFile.patch
                Merge-7-replicationMonitor.patch
                Merge-6-locatedStripedBlock.patch
                Merge-5-blockPlacementPolicies.patch
                Merge-4-blockmanagement.patch
                Merge-3-blockInfo.patch
                Merge-2-ecZones.patch
                Merge-1-codec.patch

Attaching sub-patches 1~8 from the consolidated HDFS-7285 patch as background of this proposed
change. Note that they are trunk-based. 

The root of this proposed change is the new {{BlockInfo}} hierarchy introduced by HDFS-8499.
Based on that, we are able to simplify striped blocks handling in several perspectives:
# {{BlockUCContiguous}} and {{BlockInfoUCStriped}} can now share the UC logic. This can be
seen in {{Merge-3-blockInfo.patch}}. That sub-patch also separates out a new {{StripedBlockStorageOp}}
class to handle striping-specific logic.
# By keeping {{BlockInfoUC}} as a subclass of {{BlockInfo}} (instead of a separate interface
as in the branch), we can avoid changing trunk code that relies on this relationship. This
can be seen in {{Merge-4-blockmanagement.patch}} ("Allocate and manage striped blocks in NameNode
blockmanagement module."). With this change and other pre-merge efforts (HDFS-8487, HDFS-8608,
HDFS-8623), the [rebased consolidated patch | https://issues.apache.org/jira/secure/attachment/12744101/Consolidated-20150707.patch]
has much fewer intrusive changes to {{blockmanagement}} and {{namenode}} modules than the
current HDFS-7285 consolidated patch:
{code}
Current HDFS-7285: 
2532 insertions(+), 1156 deletions(-) in blockmanagement
1826 insertions(+), 444 deletions(-) in namenode

Rebased (including this proposed simplification):
1251 insertions(+), 201 deletions(-) in blockmanagement
1324 insertions(+), 168 deletions(-) in namenode
{code}
# As we [discussed | https://issues.apache.org/jira/browse/HDFS-7285?focusedCommentId=14600362&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14600362]
under HDFS-7285, this new {{BlockInfo}} hierarchy also requires us to unify contiguous and
striped blocks handling along the direction of HDFS-8058. I think we should also take this
chance to reach a conclusion on this issue. My current thought is that we can take the HDFS-8058
approach to avoid duplicating code, and thoroughly address type safety as a follow-on (some
thoughts can be found on HDFS-8655).

The attached sub-patches 1, 2, 5, 6, 7 are mostly the same as the branch; 3, 4, 8 contain
the proposed simplification and should be more thoroughly reviewed ([~andrew.wang]: we went
over these ideas in offline discussions; could you take a look at the patches?). Per Jing's
suggestion I'm also working on generating a patch against the current HDFS-7285 branch.

> Erasure coding: revisit and simplify BlockInfoStriped and INodeFile
> -------------------------------------------------------------------
>
>                 Key: HDFS-8728
>                 URL: https://issues.apache.org/jira/browse/HDFS-8728
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: Merge-1-codec.patch, Merge-2-ecZones.patch, Merge-3-blockInfo.patch,
Merge-4-blockmanagement.patch, Merge-5-blockPlacementPolicies.patch, Merge-6-locatedStripedBlock.patch,
Merge-7-replicationMonitor.patch, Merge-8-inodeFile.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message