hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brad C" <bradmailingl...@gmail.com>
Subject Re: Hadoop Distributed Virtualisation
Date Fri, 06 Jun 2008 16:19:30 GMT
Hi Colin,

I think this would work as a lab setup for testing how hadoop handles
hardware failures but I was thinking of the opposite on a much larger
scale.

It is though by many IT individuals that its smart to take one high
end system and split into multiple machines, though few people think
of taking multiple machines and integrating them into one and then
virtualising them apart again. This has been done though end data
storage has almost always resided on a SAN which has a cost
implication. I know Hadoops Distributed Filesystem is geared towards
batch processing and larger data sets with higher latency, though has
anyone every tried integrating it with a virtual server tech like kvm?
The other possibility is that Im nuts and have completely lost the plot.

Kind Regards

Brad

On Fri, Jun 6, 2008 at 5:03 PM, Colin Freas <colinfreas@gmail.com> wrote:
> I've wondered about this using single or dual quad-core machines with one
> spindle per core, and partitioning them out into 2, 4, 8, whatever virtual
> machines, possibly marking each physical box as a "rack".
>
> There would be some initial and ongoing sysadmin costs.  But could this
> increase thoughput on a small cluster, of 2 or 3 boxes with 16 or 24 cores,
> with many jobs by limiting the number of cores each job runs on, to say 8?
> Has anyone tried such a setup?
>
>
> On Fri, Jun 6, 2008 at 10:30 AM, Brad C <bradmailinglist@gmail.com> wrote:
>
>> Hello Everyone,
>>
>> I've been brainstorming recently and its always been in the back of my
>> mind, hadoop offers the functionality of clustering comodity systems
>> together, but how would one go about virtualising them apart again?
>>
>> Kind Regards
>>
>> Brad :)
>>
>

Mime
View raw message