Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Barry, Sean F" <sean.f.barry@intel.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Subject: RE: multiple nodes one machine
Thread-Topic: multiple nodes one machine
Thread-Index: Ac0WbOQLGflAxDKOQgKOL9xmllLdnwATU8EAAAuSdjA=
Date: Mon, 9 Apr 2012 20:22:20 +0000
Message-ID: 
 <E4B77C96967B694B8E85FBFC19F3C14A0B0FBBB4@ORSMSX103.amr.corp.intel.com>
References: 
 <E4B77C96967B694B8E85FBFC19F3C14A0B0FBB3F@ORSMSX103.amr.corp.intel.com>
 <CAOcnVr0s0C0j_yk6A461dg+o==L0D_mQ73BKAf-wai4eDUObOA@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr0s0C0j_yk6A461dg+o==L0D_mQ73BKAf-wai4eDUObOA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Harsh,=20
I am interested in adding datanodes just for testing.


I have a few more things I should have said earlier.=20
My current cluster looks like this. Which I set up exactly like tutorial ht=
tp://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-no=
de-cluster/ except I am running Suse 12.1 on both boxes.

Master - running NameNode, DataNode, TaskTracker, SecNameNode and JobTtrack=
er
------
Slave - running Datanode and TaskTracker

My (4 core) slave  machine is the one that I would like to add three additi=
onal datanodes to But when I use the run-additionalDN.sh script I get an Us=
age: java DataNode    [-rollback].

Am I supposed to run the script on my master node or slave node?

-SB


-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]=20
Sent: Monday, April 09, 2012 11:35 AM
To: common-user@hadoop.apache.org
Subject: Re: multiple nodes one machine

Barry,

Depends on what you'll be testing. If you want more daemons, then yes you n=
eed to add more nodes onto the same box (configs may be tweaked to achieve =
this). If you just want MR to provide more slots for tasks, then a specific=
 task tracker property alone may be edited.

For more daemons, see http://search-hadoop.com/m/a4klk28NUr12 and a neat co=
nfig I use for running them without too much config mess:
https://gist.github.com/2345300

For the latter, see:
http://wiki.apache.org/hadoop/FAQ#I_see_a_maximum_of_2_maps.2BAC8-reduces_s=
pawned_concurrently_on_each_TaskTracker.2C_how_do_I_increase_that.3F

Alternatively, use the hadoop-test jar provided classes:
MiniDFSCluster and MiniMRCluster which can run from with a test suite itsel=
f (With multiple threads as daemons, to simply test around with).

On Mon, Apr 9, 2012 at 9:52 PM, Barry, Sean F <sean.f.barry@intel.com> wrot=
e:
> Hi all,
>
> I currently have a 2 node cluster up and running. But now I face a new is=
sue, one of my nodes is running a Datanode and a Tasktracker on a 4 core ma=
chine and in order to do a bit of proof of concept testing I would like to =
have 4 nodes running on that particular machine. Does this mean that I woul=
d need to set that up as a pseudodistributed cluster? or do you have any ot=
her suggestions? And would I need to add 3 more datanodes and 3 more tasktr=
ackers or either or?
>
> Thanks
> -SB


--
Harsh J