hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-12072) Provide fairness between EC and non-EC recovery tasks.
Date Thu, 29 Jun 2017 23:48:00 GMT
Lei (Eddy) Xu created HDFS-12072:
------------------------------------

             Summary: Provide fairness between EC and non-EC recovery tasks.
                 Key: HDFS-12072
                 URL: https://issues.apache.org/jira/browse/HDFS-12072
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: erasure-coding
    Affects Versions: 3.0.0-alpha3
            Reporter: Lei (Eddy) Xu
            Assignee: Lei (Eddy) Xu


In {{DatanodeManager#handleHeartbeat}}, it takes up to {{maxTransfer}} reconstruction tasks
for non-EC, then if the request can not be full filled, it takes more tasks from EC reconstruction
tasks.

{code}
    List<BlockTargetPair> pendingList = nodeinfo.getReplicationCommand(
        maxTransfers);
    if (pendingList != null) {
      cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId,
          pendingList));
      maxTransfers -= pendingList.size();
    }
    // check pending erasure coding tasks
    List<BlockECReconstructionInfo> pendingECList = nodeinfo
        .getErasureCodeCommand(maxTransfers);
    if (pendingECList != null) {
      cmds.add(new BlockECReconstructionCommand(
          DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList));
    }
{code}

So on a large cluster, if there are large number of constantly non-EC reconstruction tasks,
EC reconstruction tasks do not have a chance to run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message