hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mahadev konar (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Date Thu, 08 Dec 2011 00:10:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164861#comment-13164861
] 

Mahadev konar commented on MAPREDUCE-3512:
------------------------------------------

@Vinod,
 The only issue I see with that is that we might end up having only part of the event logged
when an AM crashes. Am not sure if the history event handler can handle incomplete events.
                
> Batch jobHistory disk flushes
> -----------------------------
>
>                 Key: MAPREDUCE-3512
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>    Affects Versions: 0.23.0
>            Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history
even handler ends up with a significant backlog for tests like MAPREDUCE-3402. 
> History events could be batched up based on num records / time / TaskFinishedEvents to
reduce the number of DFS writes - with the potential drawback of having to rerun some tasks
during AM recovery.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message