hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9379) Make NNThroughputBenchmark support more than 10 numThreads
Date Thu, 05 Nov 2015 03:36:27 GMT
Mingliang Liu created HDFS-9379:

             Summary: Make NNThroughputBenchmark support more than 10 numThreads
                 Key: HDFS-9379
                 URL: https://issues.apache.org/jira/browse/HDFS-9379
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
            Reporter: Mingliang Liu
            Assignee: Mingliang Liu

Currently, the {{NNThroughputBenchmark}} relies on sorted {{datanodes}} array in the lexicographical
order of datanode's {{xferAddr}}.
* There is an assertion of datanode's {{xferAddr}} lexicographical order when filling the
{{datanodes}}, see [the code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1152].
* When searching the datanode by {{DatanodeInfo}}, it uses binary search against the {{datanodes}}
array, see [the code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1187]

In {{DatanodeID}}, the {{xferAddr}} is defined as {{host:port}}. In {{NNThroughputBenchmark}},
the port is simply _the index of the tiny datanode_ plus one.

The problem here is that, when there are more than 9 tiny datanodes ({{numThreads}}), the
lexicographical order of datanode's {{xferAddr}} will be invalid as the string value of datanode
index is not in lexicographical order any more. For example, 

{{}} is greater than {{}}. The assertion will fail and the
binary search won't work.

The simple fix is to calculate the datanode index by port directly, instead of using binary

This message was sent by Atlassian JIRA

View raw message