hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: Hadoop and Cloud computing
Date Wed, 11 Aug 2010 17:55:20 GMT

One of things that you can consider looking at is maintaining data
locality during the expansion and contraction of an on demand cluster.
Maybe the on demand cluster resizing can be done intelligently without
impacting the locality much. Ofcourse it would mean moving some data
around while the resizing happens but how can it be minimized? Add
HBase into the equation as well if you'd like.

Just an idea I had while discussing this with a colleague earlier.


Sent from my iPhone

On Aug 11, 2010, at 2:43 AM, Steve Loughran <stevel@apache.org> wrote:

> On 10/08/10 15:00, Jackob Carlsson wrote:
>> Hi,
>> I am trying to write a thesis proposal about my PhD about usage of hadoop in
>> cloud computing. I need to find some open problems in cloud computing which
>> can be addressed by hadoop. I would appreciate if somebody could help me to
>> find some topics.
>> Thanks in advance
>> Jackob
> This might be a starting point
> http://www.slideshare.net/steve_l/hadoop-and-universities
> * what do you mean by "cloud computing"; if it is VM-hosted code running on Pay-as-yo-go
Infrastructure, this is the kind of problem:
> http://www.slideshare.net/steve_l/farming-hadoop-inthecloud
> -placing VMs close to the data
> -handling failure differently (don't blacklist, kill the VM)
> -making Hadoop and its clients more adaptive to clusters where the machines are moving
around more.
> Other options
> -running Hadoop physically, but use the spare cycles/memory for other work, so the tasktrackers
must co-ordinate Hadoop work scheduling with other work
> -running Hadoop directly against the underlying filesystem of the infrastructure, instead
of HDFS.
> http://www.slideshare.net/steve_l/high-availability-hadoop
> Where are you based? If you are in the UK we could meet some time, I'll be at the opentech
event in London next month.

View raw message