hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12277) NNThroughputBenchmark should use cumulativeTime to calculate ops per second.
Date Wed, 09 Aug 2017 00:42:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119251#comment-16119251

Andrew Wang commented on HDFS-12277:

Hi Michael, thanks for working on this. I think the current cumulativeTime behavior is intentional.
If you want to add additional throughput-per-thread metrics, that'd be a fine addition.

It might also be interesting to consider histogram metrics, like YCSB. Since the NN is an
in-memory system there won't be as much variability, but this would be a good way to check.

> NNThroughputBenchmark should use cumulativeTime to calculate ops per second.
> ----------------------------------------------------------------------------
>                 Key: HDFS-12277
>                 URL: https://issues.apache.org/jira/browse/HDFS-12277
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: benchmarks, tools
>            Reporter: Michael Han
>            Assignee: Michael Han
>            Priority: Minor
>         Attachments: HDFS-12277.000.patch, HDFS-12277.001.patch
> {{NNThroughputBenchmark$getOpsPerSecond}} is using {{elapsedTime}} to calculate ops per
second. {{elapsedTime}} is the time used to measure the time from the start to the finish
of the benchmark main thread, and it's also the total execution time of the single slowest
benchmark worker thread. To measure ops per second, we should use the total execution time
of all worker threads, which is {{cumulativeTime}}. Otherwise, we could get bogus result,
e.g. given same load, the more client threads we have (controlled by {{-threads}}), the larger
ops per second the number would be, with the same NN and its server side configurations.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message