hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress calculations in MR AM
Date Fri, 23 Dec 2011 02:33:30 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175238#comment-13175238
] 

Siddharth Seth commented on MAPREDUCE-3568:
-------------------------------------------

+1, Looks good. One minor change though - TestRMContainerAllocator.FakeJob isn't used any
more and can be removed.
                
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
>                 Key: MAPREDUCE-3568
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mr-am, mrv2, performance
>    Affects Versions: 0.23.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Critical
>             Fix For: 0.23.1
>
>         Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to
the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated
by looking up of each task in a big map while we can simply make do with a scan and aggregate.
With large number of tasks, this can make a difference.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message