Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-Id: <750FD559-D9E7-45E5-B9BC-6E395808314F@wensel.net>
From: Chris K Wensel <chris@wensel.net>
To: core-user@hadoop.apache.org
In-Reply-To: <3ABF9EB3-1AC4-4D73-A4FE-C1920699D28A@wensel.net>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: Reducers spawned when mapred.reduce.tasks=0
Date: Thu, 12 Mar 2009 14:40:05 -0700
References: <3ABF9EB3-1AC4-4D73-A4FE-C1920699D28A@wensel.net>


May have found the answer, waiting on confirmation from users.

Turns out 0.19.0 and .1 instantiate the reducer class when the task is  
actually intended for job/task cleanup.

branch-0.19 looks like it resolves this issue by not instantiating the  
reducer class in this case.

I've got a workaround in the next maint release:
http://github.com/cwensel/cascading/tree/wip-1.0.5

ckw

On Mar 12, 2009, at 10:12 AM, Chris K Wensel wrote:

> Hey all
>
> Have some users reporting intermittent spawning of Reducers when the  
> job.xml shows mapred.reduce.tasks=0 in 0.19.0 and .1.
>
> This is also confirmed when jobConf is queried in the (supposedly  
> ignored) Reducer implementation.
>
> In general this issue would likely go unnoticed since the default  
> reducer is IdentityReducer.
>
> but since it should be ignored in the Mapper only case, we don't  
> bother not setting the value, and subsequently comes to ones  
> attention rather abruptly.
>
> am happy to open a JIRA, but wanted to see if anyone else is  
> experiencing this issue.
>
> note the issue seems to manifest with or without spec exec.
>
> ckw
>
> --
> Chris K Wensel
> chris@wensel.net
> http://www.cascading.org/
> http://www.scaleunlimited.com/
>

--
Chris K Wensel
chris@wensel.net
http://www.cascading.org/
http://www.scaleunlimited.com/