hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Milind.Bhandar...@emc.com>
Subject Re: Platform MapReduce - Enterprise Features
Date Mon, 19 Sep 2011 19:21:19 GMT
For those who are not aware of 6-year old history :-), Sameer, Owen and I
made a trip to Wisconsin, Madison to meet with Miron Livny, who built
Condor, exploring how matchmaking could be used with MapReduce
(pre-hadoop) in October 2005. (We believed it could be used for
locality-aware scheduling.) One thing that came out of that meeting was
that Condor folks were not ready to incorporate multi-threading, which we
felt was needed for scheduler responsiveness.

- Milind

On 9/13/11 9:46 PM, "Brian Bockelman" <bbockelm@cse.unl.edu> wrote:

>
>On Sep 13, 2011, at 7:20 AM, Steve Loughran wrote:
>
>> 
>> I missed a talk at the local university by a Platform sales rep last
>>month, though I did get to offend one of the authors of condor team
>>instead [1]. by pointing out that all grid schedulers contain a major
>>assumption: that storage access times are constant across your cluster.
>>It is if you can pay for something like GPFS, but you don't get 50TB of
>>GPFS storage for $2500, which is what adding 25*2TB SATA drives would
>>cost if you stuck them on your compute nodes; $7500 for a fully
>>replicated 50TB. That's why I'm not a fan of grid systems -cost of
>>storage and networking aren't taken into account. Then there's the
>>availablity issues with the larger filesystems, that are a topic for
>>another day.
>
>For what it's worth - I do know folks who have done (are doing) data
>locality with Condor.  Condor is wonderfully flexible, easily flexible
>enough to shoot yourself in the foot.  There was also a grad student who
>did work in allowing Condor to fire up Hadoop datanodes and job trackers
>directly.
>
>For the most part you are right though - all these systems have long
>treated nodes as individual, independent units (either because the
>systems were job-oriented, not data oriented, or because they ran at
>supercomputing centers where money was no concern).
>
>This is starting to change, but change is always frustratingly slow.  On
>the upside, we now have single Condor pools that span 80 sites around the
>globe and it is easy to have two Condor pools interoperate and exchange
>jobs.  So, each system has its own strengths and weaknesses.
>
>Brian


Mime
View raw message