hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8968) New benchmark throughput tool for striping erasure coding
Date Thu, 05 Nov 2015 14:05:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991691#comment-14991691

Rakesh R commented on HDFS-8968:

bq. I mentioned I can run ErasureCodeBenchmarkThroughput with MiniDFSCluster in a unit test.
And that's only intended to test the new tool. To really benchmark performance, user has to
deploy a real cluster, right?
Yes, I agree with you. IIUC, the proposed patch supports the real cluster bench marking. I
was thinking that we could add the {{MiniDFSCluster}} unit testing also inside the ErasureCodeBenchmarkThroughput.
BTW, whats your idea to run ErasureCodeBenchmarkThroughput with MiniDFSCluster. Are you planning
to write another new unit test class for the MiniDFSCluster?

> New benchmark throughput tool for striping erasure coding
> ---------------------------------------------------------
>                 Key: HDFS-8968
>                 URL: https://issues.apache.org/jira/browse/HDFS-8968
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Rui Li
>         Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch
> We need a new benchmark tool to measure the throughput of client writing and reading
considering cases or factors:
> * 3-replica or striping;
> * write or read, stateful read or positional read;
> * which erasure coder;
> * striping cell size;
> * concurrent readers/writers using processes or threads.
> The tool should be easy to use and better to avoid unnecessary local environment impact,
like local disk.

This message was sent by Atlassian JIRA

View raw message