hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Macdonald <cra...@dcs.gla.ac.uk>
Subject Re: Using HOD to manage a production cluster[MESSAGE NOT SCANNED]
Date Sun, 01 Feb 2009 14:08:20 GMT
Some comments inline

Edward Capriolo wrote:
> I am looking at using HOD (Hadoop On Demand) to manage a production
> cluster. After reading the documentation It seems that HOD is missing
> some things that would need to be carefully set in a production
> cluster.
HOD is meant for bringing up on-demand clusters, not forever living 
> Rack Locality:
> HOD uses the -N 5 option and starts a cluster of N nodes. There seems
> to be no way to pass specific options to them individually. How can I
> make sure the set of servers selected selected will end up in
> different racks?
There are scheduler knobs in Torque and Maui that can help you here if 
this is what you want. For instance, you can ask for a HOD job to have 
its nodes allocated in a given rack.
> data node blacklist/white list
> These are listed in a file can that file be generated?
> hadoop-env
> Can I set my these options from HOD or do I have to build them into
> the hadoop tar.
> JMX settings
> Can I set my these options from HOD or do I have to build them into
> the hadoop tar.
> Upgrade with non symmetric configurations:
> Old servers > /mnt/drive1  /mnt/drive2
> New Servers > /mnt/drive1  /mnt/drive2  /mnt/drive3
Hadoop will ignore folders that dont exist, so just state all three drives.
> Can HOD ship out different configuration files to different nodes? As
> new nodes are joining the cluster for an upgrade they may have
> different configurations then the old one.
> From reading the docs it seems like HOD is great for building on
> demand clusters, but may not be ideal for managing a single permanent
> long term cluster.
Indeed, that is the aim.
> Accidental cluster destruction.
> Sounds silly but might the wrong command take out a cluster in one
> swipe. Possibly block this feature.
> Any thoughts?

View raw message