hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GorGo <gylf...@ru.is>
Subject Re: increase number of map tasks
Date Tue, 10 Jan 2012 14:09:14 GMT


I am no expert, but you could try this. 

Your problem, I guess, is that the record reader reads multiple lines of
work (tasks) and gives to each mapper and thus if you only have a few tasks
(line of work in the input file) Hadoop will not spawn multiple mappers. 

You could try this, make each input record, in your now single input file,
an independent file (with only one line) and give as input to your job the
the directory with the files (not a single file). For me, this forced the
spawning of multiple mappers. 

There is another more correct way that forces the spawning of a map task for
each line but as I was using c++ pipes that was not an option for me.

Hope this helps

sset wrote:
> Hello,
> In hdfs we have set block size - 40bytes . Input Data set is as below
> terminated with line feed.
> data1   (5*8=40 bytes)
> data2
> ......
> .......
> data10
> But still we see only 2 map tasks spawned, should have been atleast 10 map
> tasks. Each mapper performs complex mathematical computation. Not sure how
> works internally. Line feed does not work. Even with below settings map
> tasks never goes beyound 2, any way to make this spawn 10 tasks. Basically
> it should look like compute grid - computation in parallel.
> <property>
>   <name>io.bytes.per.checksum</name>
>   <value>30</value>
>   <description>The number of bytes per checksum.  Must not be larger than
>   io.file.buffer.size.</description>
> </property>
> <property>
>   <name>dfs.block.size</name>
>    <value>30</value>
>   <description>The default block size for new files.</description>
> </property>
> <property>
>   <name>mapred.tasktracker.map.tasks.maximum</name>
>   <value>10</value>
>   <description>The maximum number of map tasks that will be run
>   simultaneously by a task tracker.
>   </description>
> </property>
> single node with high configuration -> 8 cpus and 8gb memory. Hence taking
> an example of 10 data items with line feeds. We want to utilize full power
> of machine - hence want at least 10 map tasks - each task needs to perform
> highly complex mathematical simulation.  At present it looks like file
> data is the only way to specify number of map tasks via splitsize (in
> bytes) - but I prefer some criteria like line feed or whatever.
> How do we get 10 map tasks from above configuration - pls help.
> thanks

View this message in context: http://old.nabble.com/increase-number-of-map-tasks-tp33107775p33111745.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

View raw message