hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sindhu Hosamane <sindh...@gmail.com>
Subject Re: Performance on singlenode and multinode hadoop
Date Thu, 31 Jul 2014 12:58:36 GMT
Hello ,

If i am running my experiment on a server with 2 processors (4 cores each ) .
To say it has 2 processors and 8 cores .
What would be the ideal values for mapred.tasktracker.map.tasks.maximum  and mapred.tasktracker.reduce.tasks.maximum
to get maximum performance.
Your help is very much appreciated.


Regards,
Sindhu


On 29 Jul 2014, at 18:56, Harsh J <harsh@cloudera.com> wrote:

> It isn't the DataNode that does the compute spawn/work, but the TaskTracker.
> 
> If you wanted to increase MR parallelism on a single machine, you do
> not need two DNs, nor two TTs, just higher slot capacities in your
> TT's mapred-site.xml via properties
> mapred.tasktracker.map.tasks.maximum and
> mapred.tasktracker.reduce.tasks.maximum.
> 
> On Mon, Jul 28, 2014 at 4:30 PM, sindhu hosamane <sindhuht@gmail.com> wrote:
>> Hello ,
>> 
>> i set up 2 datanodes on a single machine(ubuntu machine)  accordingly
>> mentioned in the thread
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201009.mbox/%3CA3EF3F6AF24E204B812D1D24CCC8D71A03688F76@mse16be2.mse16.exchange.ms%3E
>> 
>> Ubuntu machine has 2 processors and 8 cores. Assuming that machine is
>> powerful , i Setup 2 datanodes on that same machine.
>> 
>> Now when i run jps on that multinode hadoop , i get
>> Namenode
>> Datanode
>> Datanode
>> Jobtracker
>> Tasktracker
>> Secondary Namenode
>> 
>> The above result Shows 2 datanodes are up and running
>> 
>> Also i have a single node on that ubuntu machine as well.
>> Now when i check Performance on singlenode and multinode , both are almost
>> same.So now ,
>> How do i make sure load is being distributed on both datanodes or each
>> datanode uses different cores of the ubuntu machine.
>> 
>> (Note: i know multiple datanodes on same machine is not that advantageous ,
>> but assuming my machine is powerful ..i set it up..)
>> 
>> would appreciate any advices on this.
>> 
>> Regards,
>> Sindhu
> 
> 
> 
> -- 
> Harsh J


Mime
View raw message