hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9315) Update excess storage type list properly when del hint is picked
Date Tue, 27 Oct 2015 01:51:27 GMT
Ming Ma created HDFS-9315:
-----------------------------

             Summary: Update excess storage type list properly when del hint is picked
                 Key: HDFS-9315
                 URL: https://issues.apache.org/jira/browse/HDFS-9315
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Ming Ma


HDFS-8647 makes it easier to reason about various block placement scenarios. Here is one potential
issue where {{excessTypes}} isn't updated when {{delNodeHint}} is picked. When {{delNodeHint}}
isn't picked, the excess storage identified will be removed from excessTypes.

{noformat}
      if (useDelHint(firstOne, delNodeHintStorage, addedNodeStorage,
          moreThanOne, tmpExcessTypes)) {
        cur = delNodeHintStorage;
      } else { // regular excessive replica removal
        cur = chooseReplicaToDelete((short) expectedNumOfReplicas, moreThanOne,
            exactlyOne, tmpExcessTypes);
      }

chooseReplicaToDelete(...) {
...
    excessTypes.remove(storage.getStorageType());
}
{noformat}

It isn't clear how this can happen in real world; maybe HDFS-9314. Usually when del hint is
used, the delta between expected num replica and the actual replica is one and thus this shouldn't
cause any issue.

Still it is better to make it consistent. Each time an excess replica is picked, excessTypes
should be updated regardless whether it comes from del hint or not.

Note this issue was there prior to HDFS-8647.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message