Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com
 designates 66.249.92.168 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=MArBZWFr3UrrlCfUcpZIv/BBh04+PRmNC7URwGn6VLv0wfmFwMsRzIbz2SBEAcX0ns
         y1i++ilq227TEADjQOp5aSA4rIouIxOSIzmDQVzEk4BCD+6IY136WSrWTtdBUXDGtlQD
         L/9ZSnPh3FgdAYerMKWHm6mG9F/3c9BsnCcWM=
Message-ID: <cbbf4b570806060941j422d42eak89834add7eaab04b@mail.gmail.com>
Date: Fri, 6 Jun 2008 12:41:18 -0400
From: "Edward Capriolo" <edlinuxguru@gmail.com>
To: core-user@hadoop.apache.org
Subject: Re: Hadoop Distributed Virtualisation
In-Reply-To: <2d2102ba0806060919w42e485b8t16ed836fbf040ab7@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <2d2102ba0806060730o3b2a7eb3m68886e5cf5973480@mail.gmail.com>
	 <b27f65f70806060803k8c7f54eu9673e6deb0825117@mail.gmail.com>
	 <2d2102ba0806060919w42e485b8t16ed836fbf040ab7@mail.gmail.com>

I once asked a wise man in change of a rather large multi-datacenter
service, "Have you every considered virtualization?" He replied, "All
the CPU's here are pegged at 100%"

They may be applications for this type of processing. I have thought
about systems like this from time to time. This thinking goes in
circles. Hadoop is designed for storing and processing on different
hardware.  Virtualization lets you split a system into sub-systems.

Virtualization is great for proof of concept.
For example, I have deployed this: I installed VMware with two linux
systems on my windows host, I followed a hadoop multi-system-tutorial
running on two vmware nodes. I was able to get the word count
application working, I also confirmed that blocks were indeed being
stored on both virtual systems and that processing was being shared
via MAP/REDUCE.

The processing however was slow, of course this is the fault of
VMware. VMware has a very high emulation overhead. Xen has less
overhead. LinuxVserver and OpenVZ use software virtualization (they
have very little (almost no) overhead). Regardless of how much
overhead, overhead is overhead. Personally I find the Vmware falls
short of its promises