hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7613) Block placement policy for erasure coding groups
Date Fri, 06 Mar 2015 03:57:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349914#comment-14349914

Walter Su commented on HDFS-7613:

I'm a little confused about how FSNamesystem should call BlockPlacementPolicy when a file
is striped. I looked into HDFS-7339-008.patch, I saw:
+      numTargets = isStriped ?
+          HdfsConstants.NUM_DATA_BLOCKS + HdfsConstants.NUM_PARITY_BLOCKS :
+          pendingFile.getFileReplication();
     final DatanodeStorageInfo targets[] = getBlockManager().chooseTarget4NewBlock( 
-        src, replication, clientNode, excludedNodes, blockSize, favoredNodes,
+        src, numTargets, clientNode, excludedNodes, blockSize, favoredNodes,

from the above code, I got: FSNamesystem call BlockPlacementPolicy.chooseTarget one time,
and BlockPlacementPolicy returns all blocks' locations for the blockgroup. BlockPlacementPolicy
returns blocks' locations but not replicas'. 
It's a matter of how we are thinking of second arguments. If we treat it as blocks' locations
for many blocks, it's blocks' locations. If we treat it as replicas' locations for one block,
then it is.
I'll implement the BlockPlacementPolicyFaultTolerant for EC. And I think BlockPlacementPolicyFaultTolerant
can also work in non-EC situation.

> Block placement policy for erasure coding groups
> ------------------------------------------------
>                 Key: HDFS-7613
>                 URL: https://issues.apache.org/jira/browse/HDFS-7613
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Walter Su
> Blocks in an erasure coding group should be placed in different failure domains -- different
DataNodes at the minimum, and different racks ideally.

This message was sent by Atlassian JIRA

View raw message