incubator-mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Wilkerson <>
Subject Re: request for advice on porting Hadoop 1.0.0 to Mesos
Date Mon, 02 Jan 2012 21:27:30 GMT
So I suspected it was something like that: Hadoop and Mesos are not
orthogonal, but they are not the same, they are instead like two
vectors pointing in almost the same direction, so the system of the
two of them exhibits a kind of "numerical instability" to use a linear
algebra metaphor.  So you had to either make them parallel
(fine-grained sharing) or make them orthogonal (coarse-grained
sharing).  The patch is like a nasty change of basis matrix between
two almost parallel frames of reference.  (If you look at it the right
way, you can really literally see a change of basis matrix sometimes
in adapters between two pieces of software having different schema.)

Anyway, you are optimizing for something: maximal fine-grained
control.  However, we don't need that cool feature.  We would rather
have a feature called "I don't have to look at this stuff!"  That is,
for me I want Mesos and Hadoop to be black boxes and I want to plug
them together and have it work.  When either one of them gets
upgraded, I just want to replace the old version with the new and the
whole system still "just works".

How hard would it be for you to just stake Hadoop 1.0.0 and fit it
into Mesos in the coarse-grained way that would maximise the
orthogonality between the two.  I'm sure this is conceptually trivial,
and possibly few lines of code, the problem is that there may be
concerns I didn't think of and I'm concerned with ending up with some
weird performance bug that I don't know about because I didn't write
the 3 lines of code to foo the bar when everyone knows you have to foo
the bar each time you baz or the scheduler gets into a bad state!

What would be even more awesome would be if we could do this without
actually checking Hadoop 1.0.0 into the Mesos tree and hacking on it,
but instead just having some scripts in a directory called something
like frameworks/hadoop-1.0.0-coarse-grained that could be pointed at
an Hadoop install and then just run.  This rather would enforce re-use
/ orthogonality.

How hard would this be to do?  I am basically blocked on this.


On Mon, Jan 2, 2012 at 1:18 PM, Matei Zaharia <> wrote:
> The reason for this is that Hadoop wasn't designed to run on top of a cluster manager
-- it was designed to "own" a bunch of nodes and do its own scheduling within them. If you
wanted, you could run Hadoop on top of Mesos in a coarse-grained way, where you give it one
long-lived task on each node within which it can schedule its own work. However, the patch
we wrote lets Hadoop use Mesos in a fine-grained way, acquiring nodes only for the duration
of a task and then giving them back. This is useful if you want Hadoop to coexist with other
frameworks and share resources dynamically. Maybe we should include a version for the coarse-grained
sharing use case too; we actually had one working a while back.
> By the way, I believe the patch file will work without major changes for Hadoop 1.0,
but I'm going to ask the student who's worked with Hadoop the most recently, Charles Reiss,
because he's been looking at making it work with newer Hadoop releases.
> Matei
> On Jan 2, 2012, at 8:59 AM, Daniel Wilkerson wrote:
>> Actually, I have to say it seems a bit sub-optimal for Hadoop to
>> require a 2K+ line patch file just to work with Mesos.  I thought the
>> point of Mesos was that running another service on top of it would be
>> straightforward.  For the customer to have to know this much about the
>> internals of both just to get them to work together can't be a good
>> end-state.  It just seems that there must be an architectural
>> mis-match somewhere.
>> Daniel
>> On Sun, Jan 1, 2012 at 11:45 PM, Daniel Wilkerson
>> <> wrote:
>>> We want to run Hadoop inside mesos, but we want to run the latest
>>> version, Hadoop 1.0.0.  Therefore I was considering porting your
>>> changes to the new version of Hadoop.  Looking for mesos-related files
>>> in frameworks/hadoop-0.20.2 I find this.
>>> find . -iname "*mesos*" | grep -v '\.svn\b'
>>> ./bin/mesos-executor
>>> ./bin/mesos-jobtracker-launcher
>>> ./bin/mesos-jobtracker-runner
>>> ./hadoop-mesos.patch
>>> ./src/contrib/mesos
>>> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/
>>> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/
>>> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/
>>> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/
>>> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/
>>> This hadoop-mesos.patch file looks suspiciously useful, however at 2K
>>> lines, it is not short.  I suppose I could take Hadoop 1.0.0, apply
>>> the patch, copy over the files, and start it up.  However even if this
>>> seems to work, who knows what lurks in subtle invariants of which I
>>> know not.  At least from the Mesos side, do you have any advice?
>>> Daniel

View raw message