hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Prakash (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
Date Wed, 06 Mar 2013 00:02:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13594124#comment-13594124
] 

Ravi Prakash commented on MAPREDUCE-3685:
-----------------------------------------

This is probably more for just my reference than anything. Here's my understanding from reading
the code. This is very approximate and may be inaccurate for some cases

IntermediateMemoryToMemoryMerger - Can be toggled on / off
- Merges map outputs *from* memory *to* memory
- When is it triggered? (If at all enabled, which it isn't by default) When the number of
in memory map outputs > memToMemMergeOutputsThreshold
I am guessing this was put in on the premise that it might be faster to sort fewer number
of streams even in memory. And also we can sort perhaps while waiting to fetch.

InMemoryMerger
- Merges map outputs *from* memory *to* disk
- When is it triggered? When storing more map outputs in memory would cause to go over memory
allocated for shuffle.

                
> There are some bugs in implementation of MergeManager
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-3685
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.1
>            Reporter: anty.rao
>            Assignee: anty
>            Priority: Critical
>         Attachments: MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch,
MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch,
MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch,
MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch,
MAPREDUCE-3685.patch, MAPREDUCE-3685.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message