hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
Date Thu, 10 Sep 2015 08:15:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738375#comment-14738375
] 

Kai Zheng commented on HDFS-8968:
---------------------------------

Thanks Rui for coming up the tool. Since HDFS-8833 was in, would you rebase it? Additionally,
some comments so far:
1. Maybe you could give your general overall idea about how to benchmark the read/write throughput
as comments in the class header?
2. Regarding the following usage, I thought *seq|pos* does only make sense for *read*, so
the usage would be better to be reorganized.
{code}
+    System.out.println("ErasureCodeBenchmarkThroughput <read|write|gen|clean> "
+        + "<size in MB> <ec|rep> [num clients] [seq|pos]");
{code}
3. The output as below is nice for human readable, meanwhile would be better to also consider
how scripts to post process:
{code}
+    System.out.println(type + " " + df.format(totalDataSizeMB) +
+        " MB data takes " + df.format(duration) + " s. Total throughput: " +
+        df.format(throughput) + " MB/s.");
{code}
4. In *ReadCallable* class:
1) sequenceRead => statefulRead for consistency;
2) please deduplicate the codes in sequenceRead and positionalRead.

> New benchmark throughput tool for striping erasure coding
> ---------------------------------------------------------
>
>                 Key: HDFS-8968
>                 URL: https://issues.apache.org/jira/browse/HDFS-8968
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Rui Li
>         Attachments: HDFS-8968-HDFS-7285.1.patch
>
>
> We need a new benchmark tool to measure the throughput of client writing and reading
considering cases or factors:
> * 3-replica or striping;
> * write or read, stateful read or positional read;
> * which erasure coder;
> * striping cell size;
> * concurrent readers/writers using processes or threads.
> The tool should be easy to use and better to avoid unnecessary local environment impact,
like local disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message