hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
Date Thu, 23 Jun 2016 04:58:16 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Haibo Chen updated MAPREDUCE-6724:
----------------------------------
    Attachment: mapreduce6724.001.patch

Thanks [~jira.shegalov] a lot for reviewing the approach proposed. Uploading a patch that
implements it. 
1) The patch caps maxSingleShuffleLimit at Integer.MAX_VALUE, so that all shuffle with output
bigger than Integer.MAX_VALUE will be done on disk, avoiding the unsafe conversion
2) The reset of the patch is just refactoring MergeManagerImpl so that a no-merge MergeManager
can be created in the new test.

> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6724
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>         Attachments: mapreduce6724.001.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size to an int
to allocate an instance of InMemoryMapOutput. This results in an overflow if the requested
size is bigger than Integer.MAX_VALUE and eventually causes the reducer to fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message