hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8786) Erasure coding: DataNode should transfer striped blocks before being decommissioned
Date Tue, 08 Mar 2016 04:21:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15184373#comment-15184373
] 

Rakesh R commented on HDFS-8786:
--------------------------------

Thanks [~jingzhao] for the reviews and good comments.

Could you please give few clarifications on first comment. I will take care other comments
while preparing next patch. Thanks!

comment-2=> agreed, will update in next patch

comment-3=> agreed, will raise a follow-on jira and work on.

comment-4=> agreed, will update in next patch

 
bq. comment-1=> 1. Not all the decommissioning nodes can be later used as source nodes,
since we still need to consider DataNode's current load etc. Thus I'm not sure the calculation
is correct here. In the meanwhile, I do not think we should adjust additionalReplRequired:
there is no need to leave decommissioning nodes to the next round. Thus looks like we do not
need this change.

{{getAdditionalReplRequired()}} count/value is used while choosing the required number of
target nodes logic as shown below and these target nodes will be given to reconstruction or
replication tasks. The new calculation in my patch is adjusting the additional replication
required by ignoring the decommissioning count, so that the target nodes for these will not
be chosen later. Here the idea is, will schedule decommissioning replication task only if
there are no other cases like, under replcias or needed a rack  etc. Again ErasureCodingWork
will schedule replication task for decommissioning only if the block group has {{#hasAllInternalBlocks}}.
IIUC correctly your [previous comment|https://issues.apache.org/jira/browse/HDFS-8786?focusedCommentId=15174447&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15174447]
also pointing to this ignore the decommssioning if any other tasks exists. I think the existing
{{BlockManager#chooseSourceDatanodes}} is sufficient for source selection, am I missing anything?

{code}
case-1) 9 live replicas on 5 racks and 1 decommissioning replica
Choose 1 target for rack replication task and ignore decommissioning replication task

case-2) 7 live replicas, 1 under replica and 1 decommissioning replica
Choose 1 target for reconstruction task and ignore decommissioning replication task

case-3) 6 live replicas, 2 under replica and 1 decommissioning replica
Choose 2 target for reconstruction task and ignore decommissioning replication task

case-4) 8 live replicas and 1 decommissioning replica
Choose 1 target for decommissioning replication task

case-5) 7 live replicas and 2 decommissioning replica
Choose 2 target for decommissioning replication task
{code}

{code}
ErasureCodingWork.java:-
  void chooseTargets(BlockPlacementPolicy blockplacement,
      BlockStoragePolicySuite storagePolicySuite,
      Set<Node> excludedNodes) {
    //....
    DatanodeStorageInfo[] chosenTargets = blockplacement.chooseTarget(
        getBc().getName(), getAdditionalReplRequired(), getSrcNodes()[0],
        getLiveReplicaStorages(), false, excludedNodes,
        getBlock().getNumBytes(),
        storagePolicySuite.getPolicy(getBc().getStoragePolicyID()));
		

ReplicationWork.java:-
void chooseTargets(BlockPlacementPolicy blockplacement,
      BlockStoragePolicySuite storagePolicySuite,
      Set<Node> excludedNodes) {
	  //....
      DatanodeStorageInfo[] chosenTargets = blockplacement.chooseTarget(
          getBc().getName(), getAdditionalReplRequired(), getSrcNodes()[0],
          getLiveReplicaStorages(), false, excludedNodes,
          getBlock().getNumBytes(),
          storagePolicySuite.getPolicy(getBc().getStoragePolicyID()));	
{code}


> Erasure coding: DataNode should transfer striped blocks before being decommissioned
> -----------------------------------------------------------------------------------
>
>                 Key: HDFS-8786
>                 URL: https://issues.apache.org/jira/browse/HDFS-8786
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Rakesh R
>         Attachments: HDFS-8786-001.patch, HDFS-8786-002.patch, HDFS-8786-003.patch, HDFS-8786-004.patch,
HDFS-8786-draft.patch
>
>
> Per [discussion | https://issues.apache.org/jira/browse/HDFS-8697?focusedCommentId=14609004&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14609004]
under HDFS-8697, it's too expensive to reconstruct block groups for decomm purpose.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message