hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-12072) Provide fairness between EC and non-EC recovery tasks.
Date Thu, 29 Jun 2017 23:48:00 GMT
Lei (Eddy) Xu created HDFS-12072:

             Summary: Provide fairness between EC and non-EC recovery tasks.
                 Key: HDFS-12072
                 URL: https://issues.apache.org/jira/browse/HDFS-12072
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: erasure-coding
    Affects Versions: 3.0.0-alpha3
            Reporter: Lei (Eddy) Xu
            Assignee: Lei (Eddy) Xu

In {{DatanodeManager#handleHeartbeat}}, it takes up to {{maxTransfer}} reconstruction tasks
for non-EC, then if the request can not be full filled, it takes more tasks from EC reconstruction

    List<BlockTargetPair> pendingList = nodeinfo.getReplicationCommand(
    if (pendingList != null) {
      cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId,
      maxTransfers -= pendingList.size();
    // check pending erasure coding tasks
    List<BlockECReconstructionInfo> pendingECList = nodeinfo
    if (pendingECList != null) {
      cmds.add(new BlockECReconstructionCommand(

So on a large cluster, if there are large number of constantly non-EC reconstruction tasks,
EC reconstruction tasks do not have a chance to run.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message