you can set *mapred.max.split.size* property in mapred-site.xml to create
more splits and map tasks.
On Mon, Jan 18, 2010 at 12:51 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Hello,
>
> I read the documentation about running multiple Mapper tasks, but I can't
> get multiple Mappers to work. I am running under EC2 with 10 nodes.
>
> Here's what I know:
>
> 1) I guess, by default, No. of Mapper tasks will be decided by DFS block
> size, but I would like to override that. My file is small, but each line
> triggers fairly long running complicated calculations that should be run in
> parallel.
>
> 2) I tried setting the following property in the mapred-site.xml (only on
> Master), but that doesn't seem to help:
>
> <property>
> <name>mapred.map.tasks</name>
> <value>10</value>
> </property>
>
> I still see the following message:
>
> 10/01/18 01:56:34 INFO mapred.JobClient: Launched map tasks=1
> 10/01/18 01:56:34 INFO mapred.JobClient: Data-local map tasks=1
>
> (Also, I know for fact that multiple mappers are not running!)
>
>
> 3) I read somewhere that JobConf has a method called setNumMapTasks, but
> this class has been deprecated, and as such I am not using. Besides this
> method just provides a hint to Hadoop, I heard.
>
> So how do I trigger multiple Mapper tasks? Please let me know. Thanks.
>
--
Thanks & Regards,
Chandra Prakash Bhagtani,
Impetus Infotech (india) Pvt Ltd.
|