hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chaitanya krishna" <chaitanyavv.ii...@gmail.com>
Subject Re: Help: How to change number of mappers in Hadoop streaming?
Date Sun, 26 Oct 2008 10:59:43 GMT

 In order to have different number of map tasks for each of the jobs, in the
run method of the code , I had the following syntax:

conf.setNumMapTasks(num); // for number of map tasks

conf.setNumReduceTasks(num); // for number of reduce tasks

conf is the JobConf object and num is the number of map tasks/reduce tasks
that are to be had for the job. It worked for me, hope it works for you. :)

btw, i'm using hadoop-0.17.0. I'm not sure if there is any deprecation of
these methods in later versions.

V.V.Chaitanya Krishna

On Fri, Oct 17, 2008 at 1:25 AM, Steve Gao <steve.gao@yahoo.com> wrote:

> Would anybody help me?
> Can I use
> -jobconf mapred.map.task=50 in streaming command to change the job's number
> of mappers?
> I don't have a hadoop at hand and can not verify it. Thanks for your help.
> --- On Wed, 10/15/08, Steve Gao <steve.gao@yahoo.com> wrote:
> From: Steve Gao <steve.gao@yahoo.com>
> Subject: How to change number of mappers in Hadoop streaming?
> To: core-user@hadoop.apache.org
> Cc: core-dev@hadoop.apache.org
> Date: Wednesday, October 15, 2008, 7:25 PM
> Is there a way to change number of mappers in Hadoop streaming command
> line?
> I know I can change hadoop-default.xml:
> <property>
>   <name>mapred.map.tasks</name>
>   <value>10</value>
>   <description>The default number of map tasks per job.  Typically set
>   to a prime several times greater than number of available hosts.
>   Ignored when mapred.job.tracker is "local".
>   </description>
> </property>
> But that's for all jobs. What if I just want each job has different
> NUM_OF_Mappers themselves? Thanks

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message