hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9379) Make NNThroughputBenchmark support more than 10 numThreads
Date Thu, 05 Nov 2015 03:36:27 GMT
Mingliang Liu created HDFS-9379:
-----------------------------------

             Summary: Make NNThroughputBenchmark support more than 10 numThreads
                 Key: HDFS-9379
                 URL: https://issues.apache.org/jira/browse/HDFS-9379
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode
            Reporter: Mingliang Liu
            Assignee: Mingliang Liu


Currently, the {{NNThroughputBenchmark}} relies on sorted {{datanodes}} array in the lexicographical
order of datanode's {{xferAddr}}.
* There is an assertion of datanode's {{xferAddr}} lexicographical order when filling the
{{datanodes}}, see [the code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1152].
* When searching the datanode by {{DatanodeInfo}}, it uses binary search against the {{datanodes}}
array, see [the code|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java#L1187]

In {{DatanodeID}}, the {{xferAddr}} is defined as {{host:port}}. In {{NNThroughputBenchmark}},
the port is simply _the index of the tiny datanode_ plus one.

The problem here is that, when there are more than 9 tiny datanodes ({{numThreads}}), the
lexicographical order of datanode's {{xferAddr}} will be invalid as the string value of datanode
index is not in lexicographical order any more. For example, 
{code}
...
192.168.54.40:8
192.168.54.40:9
192.168.54.40:10
192.168.54.40:11
...
{code}

{{192.168.54.40:9}} is greater than {{192.168.54.40:10}}. The assertion will fail and the
binary search won't work.

The simple fix is to calculate the datanode index by port directly, instead of using binary
search.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message