hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "mingleizhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6729) Accurately compute the test execute time in DFSIO
Date Thu, 07 Jul 2016 13:42:11 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366117#comment-15366117

mingleizhang commented on MAPREDUCE-6729:

Thanks Kai Zheng for the useful comment. And I have renamed the patch name just now.

> Accurately compute the test execute time in DFSIO
> -------------------------------------------------
>                 Key: MAPREDUCE-6729
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6729
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: benchmarks, performance, test
>            Reporter: mingleizhang
>            Assignee: mingleizhang
>            Priority: Minor
>              Labels: performance, test
>         Attachments: MAPREDUCE-6729-v1.patch.txt, MR-6729.txt
> When doing DFSIO test as a distributed i/o benchmark tool. Then especially writes plenty
of files to disk or read from, both can cause performance issue and imprecise value in a way.
The question is that existing practices needs to delete files when before running a job and
that will cause extra time consumption and furthermore cause performance issue, statistical
time error and imprecise throughput while the files are lots of. So we need to replace or
improve this hack to prevent this from happening in the future.
> {code}
> public static void testWrite() throws Exception {
>     FileSystem fs = cluster.getFileSystem();
>     long tStart = System.currentTimeMillis();
>     bench.writeTest(fs); // this line of code will cause extra time consumption because
of fs.delete(*,*) by the writeTest method
>     long execTime = System.currentTimeMillis() - tStart;
>     bench.analyzeResult(fs, TestType.TEST_TYPE_WRITE, execTime);
>   }
> private void writeTest(FileSystem fs) throws IOException {
>   Path writeDir = getWriteDir(config);
>   fs.delete(getDataDir(config), true);
>   fs.delete(writeDir, true);    
>   runIOTest(WriteMapper.class, writeDir);
>   }
> {code} 
> [https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java]

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message