hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate Carlson <natec...@natecarlson.com>
Subject Re: Sharing Hadoop slave nodes between multiple masters?
Date Wed, 26 Mar 2008 17:01:12 GMT
On Wed, 26 Mar 2008, Amar Kamat wrote:
> There are two types of slaves and 2 corresponding masters in Hadoop. The 
> 2 masters are Namenode and JobTracker while the slaves are datanodes and 
> tasktrackers resp. Each slave when started has a hardcoded master 
> information in the config that is passed to the slave during the 
> start-up. So sharing would not be possible.


That's kind of what I figured, but thought I'd ask in case someone else 
has worked on this before.

> Yes. Seems like. But be sure to have a control on the number of tasks 
> that can be run on the machine. Commonly used config is 4 maps and 4 
> reducers for a (mapred) slave that is not shared (i.e per machine). Try 
> to make sure that the total tasks that can be run simultaneously is 
> descent enough. Amar

Yeah... in most cases, the environments won't be running their slaves at 
the same time, so I'm not too worried about that. I'll see if I can work 
out a way to set hard limits though.


| nate carlson | natecars@natecarlson.com | http://www.natecarlson.com |
|       depriving some poor village of its idiot since 1981            |

View raw message