hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason <urg...@gmail.com>
Subject Re: LocalJobRunner and # of reducers
Date Mon, 02 May 2011 23:47:08 GMT
Dmitriy,

I remember I had the same problem with local jobs when I tried to
debug my multi-reducer use cases. So had to create this small patch
that resolves the issue.
You can put these classes into org.apache.hadoop.mapred package in
your local project and make sure they preceed Hadoop's jars in the
class path.

My patch is based on Cloudera 0.20.2+320 release.

Hope this helps.


On 5/2/11, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:
> Hi,
>
> i was trying to create a test based on mapreduce job in a local mode
> testing various partitioning issues.
>
> But curiously, whenever i switch mapreduce into local node, i can't
> seem to be able to configure multiple reduce tasks.
>
> Indeed, upon some investigation i found that the following fragment in
> LocalJobRunner resets all reducers to 1 :
>
> /* 177 */         int numReduceTasks = this.job.getNumReduceTasks();
> /* 178 */         if ((numReduceTasks > 1) || (numReduceTasks < 0))
> /*     */         {
> /* 180 */           numReduceTasks = 1;
> /* 181 */           this.job.setNumReduceTasks(1);
> /*     */         }
> /* 183 */         outputCommitter.setupJob(jContext);
> /* 184 */         this.status.setSetupProgress(1.0F);
> /*     */
> /* 186 */         Map mapOutputFiles = new HashMap();
> /*     */
>
>
> Is this a fundamental limitation of the local mapreduce mode? what if
> i need to write up a unit test that checks various partitioning
> functions? Is there a workaround?
>
> Also, i don't remember these problems when writing tests based on
> local mapreduce in previous versions (this is cdh3b4) , although i
> cannot be sure if i ran into exactly same situation before.
>
> thanks.
> -Dmitriy
>

Mime
View raw message