hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: LocalJobRunner and # of reducers
Date Tue, 03 May 2011 00:02:44 GMT
Thanks a bunch!

(is there any chance you could do a diff only ? )

-d

On Mon, May 2, 2011 at 4:47 PM, jason <urgisb@gmail.com> wrote:
> Dmitriy,
>
> I remember I had the same problem with local jobs when I tried to
> debug my multi-reducer use cases. So had to create this small patch
> that resolves the issue.
> You can put these classes into org.apache.hadoop.mapred package in
> your local project and make sure they preceed Hadoop's jars in the
> class path.
>
> My patch is based on Cloudera 0.20.2+320 release.
>
> Hope this helps.
>
>
> On 5/2/11, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:
>> Hi,
>>
>> i was trying to create a test based on mapreduce job in a local mode
>> testing various partitioning issues.
>>
>> But curiously, whenever i switch mapreduce into local node, i can't
>> seem to be able to configure multiple reduce tasks.
>>
>> Indeed, upon some investigation i found that the following fragment in
>> LocalJobRunner resets all reducers to 1 :
>>
>> /* 177 */         int numReduceTasks = this.job.getNumReduceTasks();
>> /* 178 */         if ((numReduceTasks > 1) || (numReduceTasks < 0))
>> /*     */         {
>> /* 180 */           numReduceTasks = 1;
>> /* 181 */           this.job.setNumReduceTasks(1);
>> /*     */         }
>> /* 183 */         outputCommitter.setupJob(jContext);
>> /* 184 */         this.status.setSetupProgress(1.0F);
>> /*     */
>> /* 186 */         Map mapOutputFiles = new HashMap();
>> /*     */
>>
>>
>> Is this a fundamental limitation of the local mapreduce mode? what if
>> i need to write up a unit test that checks various partitioning
>> functions? Is there a workaround?
>>
>> Also, i don't remember these problems when writing tests based on
>> local mapreduce in previous versions (this is cdh3b4) , although i
>> cannot be sure if i ran into exactly same situation before.
>>
>> thanks.
>> -Dmitriy
>>
>

Mime
View raw message