hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "VirtualCluster" by SteveLoughran
Date Thu, 12 Mar 2009 13:13:53 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by SteveLoughran:

The comment on the change is:
new page on bringing up virtual clusters

New page:
= Virtual Clusters =

Hadoop Clusters can be created on Virtual Machines. This works, but performance (especially
to virtualized disks) can be significantly slower than for physical machines.

== Process to set up a cluster using the Hadoop shell scripts ==

 i. All machines should be brought up to date, with the same version of Java, Hadoop, etc.

 i. All machine's(both VM's and physical machines) public key are distributed to all "~/.ssh/authorized_keys"
 i. conf/hadoop-site.xml file is similar for all the machines.
 i. /etc/hosts file must contain all the machines(VM,Physical machine) IP and Hostname.
 i. The local hostname entry in /etc/hosts must not point to or any other loopback
address (some laptop-friendly Unix distributions do this). It should be to the assigned IP
 i. conf/slaves must contain the hostname of all slaves including VM's and physical machine.
 i. conf/masters must contain only master's hostname.
 i. both conf/masters and conf/slaves files must be similar in all the participating machines.
 i. The hadoop home directory should be same for all the machines like "/root/hadoop-0.18.2/"
 i. Finally execute the Namenode -format command only in the Master machine.

Most of this configuration can be done on a single machine image, the problematic area is
host naming. Virtual networks with reverseDNS allow the hostname to be picked up from the
network. If you use this, 
 1. The namenode and job tracker hostnames should remain constant; these need separate images.
 1. Bring up other machines afterwards; use an empty conf/slaves file. 

== Trouble Areas ==

Here are things that can cause trouble.
 1. Multiple virtual network adapters. It is simpler with one network adapter/node
 1. Machines changing hostname/IPAddress on a reboot. For a long-lived virtual cluster you
need stable machine names.
 1. Pauses of an entire VM for 5-10s or longer. This happens when the virtual host is overloaded
and your VM has been swapped out. Host less VMs, or have them ask for less memory.

View raw message