hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6724) Single shuffle to memory must not exceed Integer#MAX_VALUE
Date Tue, 02 Aug 2016 07:01:20 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403505#comment-15403505
] 

Hudson commented on MAPREDUCE-6724:
-----------------------------------

SUCCESS: Integrated in Hadoop-trunk-Commit #10190 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10190/])
MAPREDUCE-6724. Single shuffle to memory must not exceed (gera: rev 6890d5b472320fa7592ed1b08b623c55a27089c6)
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManagerImpl.java


> Single shuffle to memory must not exceed Integer#MAX_VALUE
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-6724
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>             Fix For: 2.9.0
>
>         Attachments: MAPREDUCE-6724.009.patch, mapreduce6724.001.patch, mapreduce6724.002.patch,
mapreduce6724.003.patch, mapreduce6724.004.patch, mapreduce6724.005.patch, mapreduce6724.006.patch,
mapreduce6724.007.patch, mapreduce6724.008.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size to an int
to allocate an instance of InMemoryMapOutput. This results in an overflow if the requested
size is bigger than Integer.MAX_VALUE and eventually causes the reducer to fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message