hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject LocalJobRunner and # of reducers
Date Mon, 02 May 2011 22:53:19 GMT
Hi,

i was trying to create a test based on mapreduce job in a local mode
testing various partitioning issues.

But curiously, whenever i switch mapreduce into local node, i can't
seem to be able to configure multiple reduce tasks.

Indeed, upon some investigation i found that the following fragment in
LocalJobRunner resets all reducers to 1 :

/* 177 */         int numReduceTasks = this.job.getNumReduceTasks();
/* 178 */         if ((numReduceTasks > 1) || (numReduceTasks < 0))
/*     */         {
/* 180 */           numReduceTasks = 1;
/* 181 */           this.job.setNumReduceTasks(1);
/*     */         }
/* 183 */         outputCommitter.setupJob(jContext);
/* 184 */         this.status.setSetupProgress(1.0F);
/*     */
/* 186 */         Map mapOutputFiles = new HashMap();
/*     */


Is this a fundamental limitation of the local mapreduce mode? what if
i need to write up a unit test that checks various partitioning
functions? Is there a workaround?

Also, i don't remember these problems when writing tests based on
local mapreduce in previous versions (this is cdh3b4) , although i
cannot be sure if i ran into exactly same situation before.

thanks.
-Dmitriy

Mime
View raw message