mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Traiano Welcome <trai...@gmail.com>
Subject Re: How to deploy Hadoop on Mesos
Date Thu, 27 Jul 2017 12:54:34 GMT
Hi Stephen


On Thu, Jul 27, 2017 at 12:19 PM, Stephen Gran <stephen.gran@piksel.com>
wrote:

> Hi,
>
> So typically people run two sorts of workloads on hadoop -
> ad-hoc/scheduled batch work, and stream workloads (spark, flink, etc.).
>
>

I'm definitely sure we'll be using hadoop for batch workloads.
We will integrate Spark with mesos for streaming workloads.




> Both spark and flink integrate natively with mesos, so no need for an
> intermediate yarn layer.  For batch work, we're looking at the aurora
> project for job scheduling.
>
>

I haven't looked at Aurora before - would you consider it a drop in
replacement for hadoop for distributed batch workloads?



> hadoop brings some interesting things, but I've not found integration
> with mesos to ever be pain-free, so we're moving to other tools instead
> of continuing down the path of trying to get hadoop working with mesos.
>
>

Understandably :-) I think I might take your advice here. Even if a one
time integration of hadoop and mesos was successful, the pain of having to
keep the integration functional over time through rapid updates and code
changes between two unrelated project codebases would be a nightmare.


Good luck!
>
> On 27/07/17 08:50, Traiano Welcome wrote:
> > Hi Stephen
> >
> >
> > On Wed, Jul 26, 2017 at 5:18 PM, Stephen Gran <stephen.gran@piksel.com
> > <mailto:stephen.gran@piksel.com>> wrote:
> >
> >     Hi,
> >
> >     It is having discussions about whether to stop, as it's having
> trouble
> >     getting enough contributors.
> >
> >     I guess I'd ask what you need to run on hadoop, why you're looking at
> >     mesos, and then see what else is in that space.
> >
> >
> >
> > I don't know what we'd need to run on hadoop at this point - it's open
> > ended, and for our developers to decide. However, should this make a
> > difference?
> >
> > We have mesos in place as a resource scheduler for a number of
> > frameworks and would like to resource manage it using the same
> > semantics, tools and mechanisms mesos provides.
> >
> > I've looked at two books so far that show how this is done, so it seems
> > this way of managing hadoop is in use in places (ref: "Apache Mesos
> > Essentials", "Mastering Mesos"), however these books are probably out of
> > date because the procedure they describe for integrating mesos and
> > hadoop is broken.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >     Cheers,
> >
> >     On 26/07/17 14:13, Brandon Gulla wrote:
> >     > Have you looked into Apache Myriad?
> >     >
> >     > http://myriad.apache.org/
> >     >
> >     > On Wed, Jul 26, 2017 at 4:12 AM, Traiano Welcome <
> traiano@gmail.com <mailto:traiano@gmail.com>
> >     > <mailto:traiano@gmail.com <mailto:traiano@gmail.com>>> wrote:
> >     >
> >     >     Hi
> >     >
> >     >     Would anyone know of some reliable guides to deploying  apache
> >     >     hadoop on top of the mesos scheduler?
> >     >
> >     >     Thanks,
> >     >     Traiano
> >     >
> >     >
> >     >
> >     >
> >     > --
> >     > Brandon
> >
> >     --
> >     Stephen Gran
> >     Senior Technical Architect
> >
> >     picture the possibilities | piksel.com <http://piksel.com>
> >
> >
>
> --
> Stephen Gran
> Senior Technical Architect
>
> picture the possibilities | piksel.com
>

Mime
View raw message