hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Young Kim <juneng...@gmail.com>
Subject Re: Selecting only few slaves in the cluster
Date Wed, 16 Feb 2011 05:21:30 GMT
you can use a fair-scheduler library to use only some parts of nodes you 
have to run a job.

by using max/min map/reduce job counts.

here is the documentation you can reference.

http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html

Junyoung Kim (juneng603@gmail.com)
On 02/16/2011 06:33 AM, praveen.peddi@nokia.com wrote:
> Hello all,
> We have a 100 node hadoop cluster that is used for multiple purposes. I want to run few
mapred jobs and I know 4 to 5 slaves should be enough. Is there anyway to restrict my jobs
to use only 4 slaves instead of all 100. I noticed that more the number of slaves more overhead
there is.
>
> Also can I pass in hadoop parameters like mapred.child.java.opts so that the actual child
processes gets the specified value for max heap size. I want to set the heap size to 2G instead
of going with the default..
>
> Thanks
> Praveen
>

Mime
View raw message