hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris K Wensel <ch...@wensel.net>
Subject Re: Reducers spawned when mapred.reduce.tasks=0
Date Thu, 12 Mar 2009 21:40:05 GMT

May have found the answer, waiting on confirmation from users.

Turns out 0.19.0 and .1 instantiate the reducer class when the task is  
actually intended for job/task cleanup.

branch-0.19 looks like it resolves this issue by not instantiating the  
reducer class in this case.

I've got a workaround in the next maint release:


On Mar 12, 2009, at 10:12 AM, Chris K Wensel wrote:

> Hey all
> Have some users reporting intermittent spawning of Reducers when the  
> job.xml shows mapred.reduce.tasks=0 in 0.19.0 and .1.
> This is also confirmed when jobConf is queried in the (supposedly  
> ignored) Reducer implementation.
> In general this issue would likely go unnoticed since the default  
> reducer is IdentityReducer.
> but since it should be ignored in the Mapper only case, we don't  
> bother not setting the value, and subsequently comes to ones  
> attention rather abruptly.
> am happy to open a JIRA, but wanted to see if anyone else is  
> experiencing this issue.
> note the issue seems to manifest with or without spec exec.
> ckw
> --
> Chris K Wensel
> chris@wensel.net
> http://www.cascading.org/
> http://www.scaleunlimited.com/

Chris K Wensel

View raw message