hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhihai Xu <zhi...@uber.com>
Subject Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services and Apache Slider
Date Tue, 19 Jul 2016 15:12:19 GMT
+1. Looking forward to this new feature,this will open a door for Yarn to
support more use cases.

zhihai


On Tue, Jul 19, 2016 at 7:12 AM, Billie Rinaldi <billie.rinaldi@gmail.com>
wrote:

> +1. I think making Slider the basis of a services framework in YARN will
> allow us to significantly improve YARN's support for services.
>
> On Thu, Jul 14, 2016 at 7:36 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org
> > wrote:
>
> > Hi, Hadoop YARN community!
> >
> > (Cross-posting across Hadoop and Slider communities)
> >
> > I opened a JIRA a little while ago to pursue a native YARN framework for
> > services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> > It is part of a bigger effort that a bunch of us YARN community members
> > are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> and
> > first-class support for services in YARN.
> >
> > The idea is that with our current attention on making services
> > first­-class, it's time to take a fresh look at how we can make Apache
> > Hadoop YARN support services well out of the box. I’ve been looking at
> > various possibilities - ranging from a custom new framework room scratch
> to
> > using one of the existing projects - and stopped at Apache Slider (
> > http://slider.incubator.apache.org) given its association with some of
> > the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> > myself etc.).
> >
> > Slider client & AM already handles a great deal of the functionality that
> > we need. I posit that assimilating the client, ApplicationMaster etc of
> an
> > existing framework like Apache Slider can serve our purpose really well.
> My
> > early informal discussions about this with few Hadoop and Slider
> community
> > members yielded generally favourable feedback.
> >
> > The Apache Slider incubator community also discussed this and expressed
> > generally positive interest in YARN taking up Slider’s key pieces, you
> can
> > see that discussion here: https://s.apache.org/0hoh.
> >
> > So in summary, we are looking to the following
> >
> > - Code
> >    — ‘Graduate' key pieces (Slider client, AM) of Apache Slider into
> > Apache Hadoop for providing a native services experiences in YARN
> >    — Leave for now some of the pieces behind in Apache Slider - (a)
> Slider
> > agent as we won’t need it, (b) Slider packages that need more
> deliberation
> > in terms of where they will live in the long term.
> >    — Create a branch in YARN, copy this code over into a new module(s),
> > and work towards completing a functioning app running on YARN.
> >
> > - Communities & releases
> >    — Good thing is that many of Apache Slider community members are
> > already seasoned folk in the Apache Hadoop ecosystem projects. For those
> > committers & PMC in Slider that are not yet Hadoop committers / PMC,
> > without complicating things much, a proposed path forward is active
> > participation in the branch (as branch committers?) and eventually in
> > mainline YARN and thus go through a natural progression to committership
> /
> > PMC. Given that most of the members are stalwarts in the Apache
> > communities, this should be a cinch IMO.
> >    — The work on this new code can start, and depending on its state, and
> > assuming that the experiment succeeds, can be merged into trunk and later
> > picked up in the next nearest & feasible Apache Hadoop release.
> >    — While the work on forked-over-code goes on till we have a
> functioning
> > app, the current Apache Slider project continues to live, with supporting
> > releases etc. At some point in the future, when things become clearer,
> more
> > decisions can be taken on the (parts left behind in the) incubating
> > project’s future.
> >
> > Thoughts?
> >
> > Thanks
> > +Vinod
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message