hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris K Wensel <ch...@wensel.net>
Subject Re: Reducers spawned when mapred.reduce.tasks=0
Date Thu, 12 Mar 2009 21:40:05 GMT

May have found the answer, waiting on confirmation from users.

Turns out 0.19.0 and .1 instantiate the reducer class when the task is  
actually intended for job/task cleanup.

branch-0.19 looks like it resolves this issue by not instantiating the  
reducer class in this case.

I've got a workaround in the next maint release:
http://github.com/cwensel/cascading/tree/wip-1.0.5

ckw

On Mar 12, 2009, at 10:12 AM, Chris K Wensel wrote:

> Hey all
>
> Have some users reporting intermittent spawning of Reducers when the  
> job.xml shows mapred.reduce.tasks=0 in 0.19.0 and .1.
>
> This is also confirmed when jobConf is queried in the (supposedly  
> ignored) Reducer implementation.
>
> In general this issue would likely go unnoticed since the default  
> reducer is IdentityReducer.
>
> but since it should be ignored in the Mapper only case, we don't  
> bother not setting the value, and subsequently comes to ones  
> attention rather abruptly.
>
> am happy to open a JIRA, but wanted to see if anyone else is  
> experiencing this issue.
>
> note the issue seems to manifest with or without spec exec.
>
> ckw
>
> --
> Chris K Wensel
> chris@wensel.net
> http://www.cascading.org/
> http://www.scaleunlimited.com/
>

--
Chris K Wensel
chris@wensel.net
http://www.cascading.org/
http://www.scaleunlimited.com/


Mime
View raw message