Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F3DDC1182A for ; Thu, 31 Jul 2014 13:12:07 +0000 (UTC) Received: (qmail 23366 invoked by uid 500); 31 Jul 2014 13:11:59 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 23238 invoked by uid 500); 31 Jul 2014 13:11:59 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 23223 invoked by uid 99); 31 Jul 2014 13:11:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Jul 2014 13:11:59 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nitinpawar432@gmail.com designates 209.85.160.173 as permitted sender) Received: from [209.85.160.173] (HELO mail-yk0-f173.google.com) (209.85.160.173) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Jul 2014 13:11:57 +0000 Received: by mail-yk0-f173.google.com with SMTP id 131so1524016ykp.32 for ; Thu, 31 Jul 2014 06:11:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=8X4nQNWiE79dNja6pArnodz7D79kSgEF/zN1eWwwYCk=; b=jtdAwf/FSHTrEU51dA77P9LS923+w48K1LkNjOi80dGcJOcaYnFDyWtcdS585aYF0k V4wtcjPRht1+y5Adt8FzUM3VAUSz8dRRIj+pHkEvsuiNAfFV1Q0B6m1jF0jdTLCPQF3r 7anwUevV6D+1zV10YeYzEf5ljMiJWP28hrBPAtlazatBDYdbUK05I/4X9pb/xi/YHXve Yno9N3TAFIJ+Ww4cKn6sUAucYxjZ9i5UFhi3MVNuBe4UOqEYVb3OBxbJrlIE2mFQJv2m d1leJjCvIwXaEisoHJqcRViTi3uXTIvDH131c3DqiULm3P1IYNV8k5rN4Ki6uSnzKb2E Em0Q== MIME-Version: 1.0 X-Received: by 10.236.90.199 with SMTP id e47mr1770455yhf.127.1406812291973; Thu, 31 Jul 2014 06:11:31 -0700 (PDT) Received: by 10.170.110.146 with HTTP; Thu, 31 Jul 2014 06:11:31 -0700 (PDT) In-Reply-To: References: Date: Thu, 31 Jul 2014 18:41:31 +0530 Message-ID: Subject: Re: Performance on singlenode and multinode hadoop From: Nitin Pawar To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=20cf3005ddc8edc8ea04ff7d0207 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3005ddc8edc8ea04ff7d0207 Content-Type: text/plain; charset=UTF-8 what kind of jobs your tasks will be doing? are they CPU intensive or only memory intensive ? On Thu, Jul 31, 2014 at 6:28 PM, Sindhu Hosamane wrote: > Hello , > > If i am running my experiment on a server with 2 processors (4 cores each > ) . > To say it has 2 processors and 8 cores . > What would be the ideal values for mapred.tasktracker.map.tasks.maximum > and mapred.tasktracker.reduce.tasks.maximum to get maximum performance. > Your help is very much appreciated. > > > Regards, > Sindhu > > > On 29 Jul 2014, at 18:56, Harsh J wrote: > > > It isn't the DataNode that does the compute spawn/work, but the > TaskTracker. > > > > If you wanted to increase MR parallelism on a single machine, you do > > not need two DNs, nor two TTs, just higher slot capacities in your > > TT's mapred-site.xml via properties > > mapred.tasktracker.map.tasks.maximum and > > mapred.tasktracker.reduce.tasks.maximum. > > > > On Mon, Jul 28, 2014 at 4:30 PM, sindhu hosamane > wrote: > >> Hello , > >> > >> i set up 2 datanodes on a single machine(ubuntu machine) accordingly > >> mentioned in the thread > >> > http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201009.mbox/%3CA3EF3F6AF24E204B812D1D24CCC8D71A03688F76@mse16be2.mse16.exchange.ms%3E > >> > >> Ubuntu machine has 2 processors and 8 cores. Assuming that machine is > >> powerful , i Setup 2 datanodes on that same machine. > >> > >> Now when i run jps on that multinode hadoop , i get > >> Namenode > >> Datanode > >> Datanode > >> Jobtracker > >> Tasktracker > >> Secondary Namenode > >> > >> The above result Shows 2 datanodes are up and running > >> > >> Also i have a single node on that ubuntu machine as well. > >> Now when i check Performance on singlenode and multinode , both are > almost > >> same.So now , > >> How do i make sure load is being distributed on both datanodes or each > >> datanode uses different cores of the ubuntu machine. > >> > >> (Note: i know multiple datanodes on same machine is not that > advantageous , > >> but assuming my machine is powerful ..i set it up..) > >> > >> would appreciate any advices on this. > >> > >> Regards, > >> Sindhu > > > > > > > > -- > > Harsh J > > -- Nitin Pawar --20cf3005ddc8edc8ea04ff7d0207 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
what kind of jobs your tasks will = be doing?=C2=A0
are they CPU inte= nsive or only memory intensive ?=C2=A0


On Thu,= Jul 31, 2014 at 6:28 PM, Sindhu Hosamane <sindhuht@gmail.com> wrote:
Hello ,

If i am running my experiment on a server with 2 processors (4 cores each )= .
To say it has 2 processors and 8 cores .
What would be the ideal values for mapred.tasktracker.map.tasks.maximum =C2= =A0and mapred.tasktracker.reduce.tasks.maximum to get maximum performance.<= br> Your help is very much appreciated.


Regards,
Sindhu


On 29 Jul 2014, at 18:56, Harsh J <harsh@cloudera.com> wrote:

> It isn't the DataNod= e that does the compute spawn/work, but the TaskTracker.
>
> If you wanted to increase MR parallelism on a single machine, you do > not need two DNs, nor two TTs, just higher slot capacities in your
> TT's mapred-site.xml via properties
> mapred.tasktracker.map.tasks.maximum and
> mapred.tasktracker.reduce.tasks.maximum.
>
> On Mon, Jul 28, 2014 at 4:30 PM, sindhu hosamane <sindhuht@gmail.com> wrote:
>> Hello ,
>>
>> i set up 2 datanodes on a single machine(ubuntu machine) =C2=A0acc= ordingly
>> mentioned in the thread
>> http://mail-archives.apache.org/mod_mbox= /hadoop-common-user/201009.mbox/%3CA3EF3F6AF24E204B812D1D24CCC8D71A03688F76= @mse16be2.mse16.exchange.ms%3E
>>
>> Ubuntu machine has 2 processors and 8 cores. Assuming that machine= is
>> powerful , i Setup 2 datanodes on that same machine.
>>
>> Now when i run jps on that multinode hadoop , i get
>> Namenode
>> Datanode
>> Datanode
>> Jobtracker
>> Tasktracker
>> Secondary Namenode
>>
>> The above result Shows 2 datanodes are up and running
>>
>> Also i have a single node on that ubuntu machine as well.
>> Now when i check Performance on singlenode and multinode , both ar= e almost
>> same.So now ,
>> How do i make sure load is being distributed on both datanodes or = each
>> datanode uses different cores of the ubuntu machine.
>>
>> (Note: i know multiple datanodes on same machine is not that advan= tageous ,
>> but assuming my machine is powerful ..i set it up..)
>>
>> would appreciate any advices on this.
>>
>> Regards,
>> Sindhu
>
>
>
> --
> Harsh J




--
= Nitin Pawar
--20cf3005ddc8edc8ea04ff7d0207--