hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2722) Gridmix simulated job's map's hdfsBytesRead counter is wrong when compressed input is used
Date Fri, 02 Mar 2012 06:04:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220701#comment-13220701
] 

Amar Kamat commented on MAPREDUCE-2722:
---------------------------------------

Ravi, compression-emulation is a feature having 3 parts
# Input compression emulation
# Intermediate compression emulation
# Output compression emulation

Intermediate and output compression emulation happens only when the compression-emulation
feature is turned on and the job's config has those parameters set.
For input compression, Gridmix relies on 'mapred.input.dir'. If there are compressed input
files only then input compression emulation will be attempted.

Scale the input-data-size field only if input-compression-emulation is desired.
                
> Gridmix simulated job's map's hdfsBytesRead counter is wrong when compressed input is
used
> ------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2722
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2722
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/gridmix
>            Reporter: Ravi Gummadi
>            Assignee: Ravi Gummadi
>         Attachments: 2722.v1.patch, MR2722.patch
>
>
> When compressed input was used by original job's map task, then the simulated job's map
task's hdfsBytesRead counter is wrong if compression emulation is enabled. This issue is because
hdfsBytesRead of map task of original job is considered as uncompressed map input size by
Gridmix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message