Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-Id: <04E5698B-23E2-47B2-B62B-1F22E10E8FBE@wensel.net>
From: Chris K Wensel <chris@wensel.net>
To: core-user@hadoop.apache.org
In-Reply-To: <49B9D0E4.6020902@yahoo-inc.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: Reducers spawned when mapred.reduce.tasks=0
Date: Fri, 13 Mar 2009 17:22:31 -0700
References: <3ABF9EB3-1AC4-4D73-A4FE-C1920699D28A@wensel.net>
 <750FD559-D9E7-45E5-B9BC-6E395808314F@wensel.net>
 <49B9D0E4.6020902@yahoo-inc.com>

fwiw, we have released a workaround for this issue in Cascading 1.0.5.

http://www.cascading.org/
http://cascading.googlecode.com/files/cascading-1.0.5.tgz

In short, Hadoop 0.19.0 and .1 instantiate the users Reducer class and  
subsequently calls configure() when there is no intention to use the  
class (during job/task cleanup tasks).

This clearly can cause havoc for users who use configure() to  
initialize resources used by the reduce() method.

Testing for jobConf.getNumReduceTasks() is 0 inside the configure()  
method seems to work out well.

branch-0.19 looks like it won't instantiate the Reducer class during  
job/task cleanup tasks, so I expect will leak into future releases.

cheers,

ckw

On Mar 12, 2009, at 8:20 PM, Amareshwari Sriramadasu wrote:

> Are you seeing reducers getting spawned from web ui? then, it is a  
> bug.
> If not, there won't be reducers spawned, it could be job-setup/ job- 
> cleanup task that is running on a reduce slot. See HADOOP-3150 and  
> HADOOP-4261.
> -Amareshwari
> Chris K Wensel wrote:
>>
>> May have found the answer, waiting on confirmation from users.
>>
>> Turns out 0.19.0 and .1 instantiate the reducer class when the task  
>> is actually intended for job/task cleanup.
>>
>> branch-0.19 looks like it resolves this issue by not instantiating  
>> the reducer class in this case.
>>
>> I've got a workaround in the next maint release:
>> http://github.com/cwensel/cascading/tree/wip-1.0.5
>>
>> ckw
>>
>> On Mar 12, 2009, at 10:12 AM, Chris K Wensel wrote:
>>
>>> Hey all
>>>
>>> Have some users reporting intermittent spawning of Reducers when  
>>> the job.xml shows mapred.reduce.tasks=0 in 0.19.0 and .1.
>>>
>>> This is also confirmed when jobConf is queried in the (supposedly  
>>> ignored) Reducer implementation.
>>>
>>> In general this issue would likely go unnoticed since the default  
>>> reducer is IdentityReducer.
>>>
>>> but since it should be ignored in the Mapper only case, we don't  
>>> bother not setting the value, and subsequently comes to ones  
>>> attention rather abruptly.
>>>
>>> am happy to open a JIRA, but wanted to see if anyone else is  
>>> experiencing this issue.
>>>
>>> note the issue seems to manifest with or without spec exec.
>>>
>>> ckw
>>>
>>> -- 
>>> Chris K Wensel
>>> chris@wensel.net
>>> http://www.cascading.org/
>>> http://www.scaleunlimited.com/
>>>
>>
>> -- 
>> Chris K Wensel
>> chris@wensel.net
>> http://www.cascading.org/
>> http://www.scaleunlimited.com/
>>
>

--
Chris K Wensel
chris@wensel.net
http://www.cascading.org/
http://www.scaleunlimited.com/