hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8823) Move replication factor into individual blocks
Date Thu, 30 Jul 2015 05:30:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647211#comment-14647211
] 

Zhe Zhang commented on HDFS-8823:
---------------------------------

If we keep replication factor in both NN and BM layers, it seems we should consider doing
the same for {{storagePolicy}}? In that case the above mem consumption analysis should be
updated.

To give my two cents: {{replication factor}}, {{storage policy}}, and {{EC policy}} (potentially)
are the same class of control knobs for HDFS users and we should probably consolidate the
discussions on how to manage them in the context of separating NN and BM layers. 

> Move replication factor into individual blocks
> ----------------------------------------------
>
>                 Key: HDFS-8823
>                 URL: https://issues.apache.org/jira/browse/HDFS-8823
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: HDFS-8823.000.patch
>
>
> This jira proposes to record the replication factor in the {{BlockInfo}} class. The changes
have two advantages:
> * Decoupling the namespace and the block management layer. It is a prerequisite step
to move block management off the heap or to a separate process.
> * Increased flexibility on replicating blocks. Currently the replication factors of all
blocks have to be the same. The replication factors of these blocks are equal to the highest
replication factor across all snapshots. The changes will allow blocks in a file to have different
replication factor, potentially saving some space.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message