hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3512) Batch jobHistory disk flushes
Date Wed, 07 Dec 2011 23:02:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13164810#comment-13164810
] 

Siddharth Seth commented on MAPREDUCE-3512:
-------------------------------------------

bq. Perhaps what we want to do is to have a non-blocking check of the event queue so we can
batch all events currently on the queue up to a given number of events in a single write.
This way if there are not very many events we do more writes and the events are output quickly
but if we start to fall behind in the writes then we start batching them up into bigger chunks
which are more efficient.

Yep. That's a nice simple way of dealing with the queue and takes care of short/long running
tasks. 
                
> Batch jobHistory disk flushes
> -----------------------------
>
>                 Key: MAPREDUCE-3512
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>    Affects Versions: 0.23.0
>            Reporter: Siddharth Seth
>
> The mr-am flushes each individual job history event to disk for AM recovery. The history
even handler ends up with a significant backlog for tests like MAPREDUCE-3402. 
> History events could be batched up based on num records / time / TaskFinishedEvents to
reduce the number of DFS writes - with the potential drawback of having to rerun some tasks
during AM recovery.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message