hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11535) Performance analysis of new DFSNetworkTopology#chooseRandom
Date Fri, 17 Mar 2017 23:48:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930906#comment-15930906

Chen Liang commented on HDFS-11535:

Thanks [~szetszwo] for sharing this idea!

[~arpitagarwal] also just discussed with me about this. The takeaways are that:
Given that the new method takes 1.6* time as mentioned in performance tests, and the threshold-based
approach will likely always take one call, which makes it always either 1* (call old) or 1.6*
(call new) of old time in most siturations. While [~szetszwo]'s proposal always takes either
1* (call old success) or 2.6* (call old fail then call new) of current time. So the threshold-based
approach should be able to achieve a slightly better performance. But given that the time
of one call is a small value by itself already, 1.6* or 2.6* is not that a big difference.
 And they both give great improvement for skewed storage type distribution, which is the more
important of this improvement.

Also, a drawback of the threshold-based one is it adds one configuration field. It's not practical
to assume that the administrators will adjust this value, so the default will likely just
be used everywhere. However this 0.6 threshold is concluded from the current performance test,
so the value may change (e.g. code change, NN performance). Picking the right value can become
tricky over time.

In short, this appears to me as a tradeoff of optimal performance v.s. administrative overhead/feasibility.
Based on the discussion, we have agreed on that [~szetszwo]'s two-trial approach should be
the slightly better approach.

> Performance analysis of new DFSNetworkTopology#chooseRandom
> -----------------------------------------------------------
>                 Key: HDFS-11535
>                 URL: https://issues.apache.org/jira/browse/HDFS-11535
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Chen Liang
>            Assignee: Chen Liang
>         Attachments: HDFS-11535.001.patch, PerfTest.pdf
> This JIRA is created to post the results of some performance experiments we did.  For
those who are interested, please the attached .pdf file for more detail. The attached patch
file includes the experiment code we ran. 
> The key insights we got from these tests is that: although *the new method outperforms
the current one in most cases*. There is still *one case where the current one is better*.
Which is when there is only one storage type in the cluster, and we also always look for this
storage type. In this case, it is simply a waste of time to perform storage-type-based pruning,
blindly picking up a random node (current methods) would suffice.
> Therefore, based on the analysis, we propose to use a *combination of both the old and
the new methods*:
> say, we search for a node of type X, since now inner node all keep storage type info,
we can *just check root node to see if X is the only type it has*. If yes, blindly picking
a random leaf will work, so we simply call the old method, otherwise we call the new method.
> There is still at least one missing piece in this performance test, which is garbage
collection. The new method does a few more object creation when doing the search, which adds
overhead to GC. I'm still thinking of any potential optimization but this seems tricky, also
I'm not sure whether this optimization worth doing at all. Please feel free to leave any comments/suggestions.
> Thanks [~arpitagarwal] and [~szetszwo] for the offline discussion.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message