hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5125) TestDFSIO should write less compressible data
Date Tue, 14 May 2013 04:25:17 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13656711#comment-13656711
] 

Matt Foley commented on MAPREDUCE-5125:
---------------------------------------

Changed Target Version to 1.3.0 upon release of 1.2.0.  Please change to 1.2.1 if you intend
to submit a fix for branch-1.2.
                
> TestDFSIO should write less compressible data
> ---------------------------------------------
>
>                 Key: MAPREDUCE-5125
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5125
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 2.0.3-alpha, 1.1.2
>            Reporter: Todd Lipcon
>            Priority: Minor
>
> Currently, TestDFSIO writes a short repeating string of sequential (byte)0 through (byte)50.
This makes its output very compressible (I measured 250:1 by LZOing the resulting file). This
makes the results of TestDFSIO very hard to compare when running on HDFS vs other file systems
which may include some compression on the network, disk, or both -- what is ostensibly a benchmark
of IO throughput yields completely skewed results towards the system with compression.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message