mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jie Yu <yujie....@gmail.com>
Subject Re: Does Mesos support Hadoop MR V2
Date Sat, 26 Jul 2014 22:19:28 GMT
I am interested in investigating the idea of YARN on top of Mesos. One of
the benefits I can think of is that we can get rid of the static resource
allocation between YARN and Mesos clusters. In that way, Mesos can allocate
those resources that are not used by YARN to other Mesos frameworks like
Aurora, Marathon, etc, to increase the resource utilization of the entire
data center. Also, we could avoid running each MRv2 job as a framework
which I think might cause some maintenance complexity (e.g. for framework
rate limiting, etc). Finally, YARN currently does not have a good isolation
support. It only supports cpu isolation right now (using cgroups). By
porting YARN on top of Mesos, we might be able to leverage the existing
Mesos containerizer strategy to provide better isolation between tasks.
Maxime, I am curious why do you think it does not make sense to run YARN
over Mesos? Since I am not super familar with YARN, I might be missing
something.

I have been thinking of making ResourceManager in YARN a Mesos framework
and making NodeManager a Mesos executor. The NodeManager will launch
containers using primitives provided by Mesos so that we have a consistent
containerizer layer. I haven't fully figured out how this could be done yet
(e.g., nested containers, communication between NodeManager and
ResourceManager, etc.), but I would love to explore this direction. I would
like to hear about any feedback/suggestions you guys have about this
direction.

Thanks,
- Jie


On Fri, Jul 25, 2014 at 1:39 PM, Maxime Brugidou <maxime.brugidou@gmail.com>
wrote:

> We run both mesos and yarn in prod and it does not make sense to run yarn
> over mesos.
>
> However it would be interesting to find a way to run MRv2 jobs on mesos
> with some custom layer to swap yarn with mesos. Not sure how to start
> though... MRv2 contains a yarn application master that needs to be
> rewritten as a mesos framework scheduler. This is probably doable. However
> with MRv2 every map reduce job would be mapped as a new framework in Mesos.
> Not sure how many frameworks mesos can run and scale up to. Especially
> short lived frameworks.
> On Jul 25, 2014 8:54 PM, "Tom Arnfeld" <tom@duedil.com> wrote:
>
>> Hey Luyi,
>>
>> That's correct, the Hadoop framework currently only supports Hadoop 2
>> MRv1. It also doesn't have great support for the HA jobtracker available in
>> newer versions of Hadoop, but I've been working on that the past few weeks.
>>
>> I'm not sure how Hadoop 2 would play with Mesos, but very interested to
>> find out more. Am I correct in thinking MRv2 will only run on top of YARN?
>>
>> I wonder if anyone else on the mailing list is running YARN on top of
>> Mesos...
>>
>> Tom.
>>
>> On Friday, 25 July 2014, Luyi Wang <wangluyi1982@gmail.com> wrote:
>>
>>> Checked the mesos github(https://github.com/mesos/hadoop). It listed
>>> support for MapReduce V1
>>>
>>> How about the MR V2?
>>>
>>> Right now we are using cloudera to manage hadoop clusters where uses
>>> MRV2. We are planning to migrate all our services to mesos(still in the
>>> initial investigating stage).  Good suggestions, advice and experiences are
>>> welcomed.
>>>
>>> Thanks a lot!
>>>
>>>
>>> -Luyi.
>>>
>>>
>>>
>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message