Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 12 Nov 2015 19:19:11 +0000 (UTC)
From: "Zhe Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12859501.1440643520000.53273.1447355951059@Atlassian.JIRA>
In-Reply-To: <JIRA.12859501.1440643520000@Atlassian.JIRA>
References: <JIRA.12859501.1440643520000@Atlassian.JIRA>
 <JIRA.12859501.1440643520144@arcas>
Subject: [jira] [Commented] (HDFS-8968) New benchmark throughput tool for
 striping erasure coding
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-8968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15002690#comment-15002690 ] 

Zhe Zhang commented on HDFS-8968:
---------------------------------

Great work Rui! The latest patch LGTM (thanks for adding the UT!).  +1 and I will commit it to trunk shortly.

I see the following items that we can do as follow-ons:
*Tests to add*
# Pread mode can add support for reading from random file offsets
# We can test {{seek}} in non-positional read
# We can test different buffer sizes in reading
# Similarly, small writes is also an important scenario to test
# Another important test case is slow writer (e.g., Flume). We can add delays between write requests

*Code optimizations*
# The last paragraph ("For example xxx") of the Javadoc in {{ErasureCodeBenchmarkThroughput}} can be converted to a real example of commands to run
# In the Javadoc we should also specify the requirement on clusters. IIUC we need to have at least 9 DNs
# Maybe also add the logic to clear OS page cache after each run? I think at least clearing client page cache is doable in {{ErasureCodeBenchmarkThroughput}}. Clearing DN page cache is harder.
# I imagine {{ErasureCodeBenchmarkThroughput}} will be used together with other scripts to automatically repeat test multiple times, interpret results etc. So {{TestErasureCodeBenchmarkThroughput}} can verify it generates expected outputs.

> New benchmark throughput tool for striping erasure coding
> ---------------------------------------------------------
>
>                 Key: HDFS-8968
>                 URL: https://issues.apache.org/jira/browse/HDFS-8968
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Rui Li
>         Attachments: HDFS-8968-HDFS-7285.1.patch, HDFS-8968-HDFS-7285.2.patch, HDFS-8968.3.patch, HDFS-8968.4.patch, HDFS-8968.5.patch
>
>
> We need a new benchmark tool to measure the throughput of client writing and reading considering cases or factors:
> * 3-replica or striping;
> * write or read, stateful read or positional read;
> * which erasure coder;
> * striping cell size;
> * concurrent readers/writers using processes or threads.
> The tool should be easy to use and better to avoid unnecessary local environment impact, like local disk.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)