hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jothi Padmanabhan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-884) TestReduceFetchFromPartialMem fails sometimes
Date Sun, 30 Aug 2009 15:45:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749278#action_12749278
] 

Jothi Padmanabhan commented on MAPREDUCE-884:
---------------------------------------------

I think I know what is happening here. All the maps are shuffled to memory, but when the last
map is shuffled, it just happens that the memory limit is reached and a memory to disk merge
is triggered and all the output touches the disk. As a result, spill counter is the same as
map output records. A simple fix would be to increase the number of maps so that the last
few maps reside in memory and so the assert condition is met.

I will fix this as a part of MAPREDUCE-157

> TestReduceFetchFromPartialMem fails sometimes
> ---------------------------------------------
>
>                 Key: MAPREDUCE-884
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-884
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: test
>            Reporter: Amar Kamat
>
> TestReduceFetchFromPartialMem failed with the following exception trace :
> {code}
> Expected some records not spilled during reduce40980)
> junit.framework.AssertionFailedError: Expected some records not spilled during reduce40980)
>         at org.apache.hadoop.mapred.TestReduceFetchFromPartialMem.testReduceFromPartialMem(TestReduceFetchFromPartialMem.java:94)
>         at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
>         at junit.extensions.TestSetup$1.protect(TestSetup.java:23)
>         at junit.extensions.TestSetup.run(TestSetup.java:27)
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message