hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gera Shegalov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
Date Wed, 20 Jul 2016 04:41:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385311#comment-15385311
] 

Gera Shegalov commented on MAPREDUCE-6724:
------------------------------------------

Hi [~haibochen], thanks for working on the patch. Initially, I probably had had another idea
with the same objective of keeping the small. If you and [~templedf] agree that the new test
version tests what's needed, I'd suggest we keep it.

> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6724
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>         Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch, mapreduce6724.003.patch,
mapreduce6724.004.patch, mapreduce6724.005.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size to an int
to allocate an instance of InMemoryMapOutput. This results in an overflow if the requested
size is bigger than Integer.MAX_VALUE and eventually causes the reducer to fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message