Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of juneng603@gmail.com designates
 74.125.83.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=V3zgVv6VjBQ7QdXONZ4bqam7thzav1D9ZRT7QTDRTf4Sou5NvM9m+ymdMDVZYO7nGO
         eXcwOf/UqDxksLgFDFZCosUyT1N+uNyGe72OJnMxXTrsJQOWSFAQBJjHicvQIV48cofv
         jQf05Ui3GGW0KqRLlwCJHfvdSfIRkiRrg8lXs=
Message-ID: <4D5B5EDA.2060808@gmail.com>
Date: Wed, 16 Feb 2011 14:21:30 +0900
From: Jun Young Kim <juneng603@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: Re: Selecting only few slaves in the cluster
References: 
 <6C870DB6C8A84F41898F05D559707F23085B5ED2@008-AM1MPN1-012.mgdnok.nokia.com>
In-Reply-To: 
 <6C870DB6C8A84F41898F05D559707F23085B5ED2@008-AM1MPN1-012.mgdnok.nokia.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

you can use a fair-scheduler library to use only some parts of nodes you 
have to run a job.

by using max/min map/reduce job counts.

here is the documentation you can reference.

http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html

Junyoung Kim (juneng603@gmail.com)
On 02/16/2011 06:33 AM, praveen.peddi@nokia.com wrote:
> Hello all,
> We have a 100 node hadoop cluster that is used for multiple purposes. I want to run few mapred jobs and I know 4 to 5 slaves should be enough. Is there anyway to restrict my jobs to use only 4 slaves instead of all 100. I noticed that more the number of slaves more overhead there is.
>
> Also can I pass in hadoop parameters like mapred.child.java.opts so that the actual child processes gets the specified value for max heap size. I want to set the heap size to 2G instead of going with the default..
>
> Thanks
> Praveen
>