hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Shuffle In Memory in 0.21.0
Date Thu, 11 Mar 2010 22:33:43 GMT
I also noticed this in 0.20.2:
        conf.getFloat("mapred.job.
shuffle.merge.percent", 0.66f);
And this in 0.21.0:
                          jobConf.getFloat(JobContext.SHUFFLE_MERGE_EPRCENT,

                                           0.90f));
I suggest changing the default limit back to 0.66 - 0.9 is too high.
Also the spelling should be SHUFFLE_MERGE_P*E*RCENT


On Thu, Mar 11, 2010 at 10:49 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> I checked out code from git://github.com/apache/hadoop-mapreduce.git which
> I assume is 0.21.0
>
> The new reserve() method at line 216 has this code:
>     if (usedMemory > memoryLimit) {
>       LOG.debug(mapId + ": Stalling shuffle since usedMemory (" +
> usedMemory +
>                ") is greater than memoryLimit (" + memoryLimit + ")");
>
>       return stallShuffle;
>     }
> The above check isn't enough because of the following scenario:
> requestedSize < maxSingleShuffleLimit
> usedMemory < memoryLimit
> usedMemory + requestedSize > memoryLimit
>
> Meaning we shouldn't call unconditionalReserve() after this check:
>     if (usedMemory > memoryLimit) {
>
> What do you think ?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message