Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
Sender: sjlee0@gmail.com
In-Reply-To: <D3B256A5.1F2D1%gsaha@hortonworks.com>
References: <0C064C3B-0ECD-443A-BBB3-277C3725E700@apache.org> <D3B256A5.1F2D1%gsaha@hortonworks.com>
From: Sangjin Lee <sjlee@apache.org>
Date: Mon, 18 Jul 2016 18:32:06 -0700
Message-ID: <CADbBEnu=X1+K_0wSmFWPCSuzWi8-8zZQwRKgLfzQwX+J3tMwSw@mail.gmail.com>
Subject: Re: [DISCUSS] YARN-5079 : Native YARN framework layer for services
 and Apache Slider
To: Gour Saha <gsaha@hortonworks.com>
Cc: "dev@slider.incubator.apache.org" <dev@slider.incubator.apache.org>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=001a113e237a9e8af40537f30e85
archived-at: Tue, 19 Jul 2016 01:32:32 -0000

--001a113e237a9e8af40537f30e85
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

+1. It would be great if it becomes the default way for services to run on
YARN. That would also make it easier to make YARN features support services
better. As you mentioned, there would be a lot of logistics to work out,
and we'll need to talk about them in more detail as they come up.

Regards,
Sangjin

On Mon, Jul 18, 2016 at 12:23 PM, Gour Saha <gsaha@hortonworks.com> wrote:

> +1 forwarding from the discussion in Slider DL
>
> Note: On the Slider side, we plan to create a branch corresponding to thi=
s
> YARN branch. In this, we intend to remove all the pieces which will move
> to the YARN branch (Slider core, AM, client). We will retain the Agent an=
d
> app-packages which will then depend on the new hadoop-slider module (in
> addition to the existing hadoop modules that it already depends on). This
> will create a single view of Slider codebase, exactly as it stands today,
> fully supporting the current app-packages. Slider can even release its
> future versions from this new branch, once the hadoop-sldier module
> reaches a matured state. It will be an easier path for existing Slider
> users/app-owners to move to the future stable state of Slider completely
> off of Hadoop YARN codebase. All that would be expected is to migrate the
> app-packages to the agent-less version. Of course the Slider community
> will do the migration for the current app-packages in the Slider codebase=
.
>
> -Gour
>
> On 7/14/16, 7:36 PM, "Vinod Kumar Vavilapalli" <vinodkv@apache.org> wrote=
:
>
> >Hi, Hadoop YARN community!
> >
> >(Cross-posting across Hadoop and Slider communities)
> >
> >I opened a JIRA a little while ago to pursue a native YARN framework for
> >services: https://issues.apache.org/jira/browse/YARN-5079.
> >
> >It is part of a bigger effort that a bunch of us YARN community members
> >are interested in making progress on: YARN-4692 - [Umbrella] Simplified
> >and first-class support for services in YARN.
> >
> >The idea is that with our current attention on making services
> >first=C2=AD-class, it's time to take a fresh look at how we can make Apa=
che
> >Hadoop YARN support services well out of the box. I=E2=80=99ve been look=
ing at
> >various possibilities - ranging from a custom new framework room scratch
> >to using one of the existing projects - and stopped at Apache Slider
> >(http://slider.incubator.apache.org) given its association with some of
> >the YARN community members (Steve Loughran, Devaraj Das, Arun C Murthy,
> >myself etc.).
> >
> >Slider client & AM already handles a great deal of the functionality tha=
t
> >we need. I posit that assimilating the client, ApplicationMaster etc of
> >an existing framework like Apache Slider can serve our purpose really
> >well. My early informal discussions about this with few Hadoop and Slide=
r
> >community members yielded generally favourable feedback.
> >
> >The Apache Slider incubator community also discussed this and expressed
> >generally positive interest in YARN taking up Slider=E2=80=99s key piece=
s, you
> >can see that discussion here: https://s.apache.org/0hoh.
> >
> >So in summary, we are looking to the following
> >
> >- Code
> >   =E2=80=95 =E2=80=98Graduate' key pieces (Slider client, AM) of Apache=
 Slider into
> >Apache Hadoop for providing a native services experiences in YARN
> >   =E2=80=95 Leave for now some of the pieces behind in Apache Slider - =
(a)
> >Slider agent as we won=E2=80=99t need it, (b) Slider packages that need =
more
> >deliberation in terms of where they will live in the long term.
> >   =E2=80=95 Create a branch in YARN, copy this code over into a new mod=
ule(s),
> >and work towards completing a functioning app running on YARN.
> >
> >- Communities & releases
> >   =E2=80=95 Good thing is that many of Apache Slider community members =
are
> >already seasoned folk in the Apache Hadoop ecosystem projects. For those
> >committers & PMC in Slider that are not yet Hadoop committers / PMC,
> >without complicating things much, a proposed path forward is active
> >participation in the branch (as branch committers?) and eventually in
> >mainline YARN and thus go through a natural progression to committership
> >/ PMC. Given that most of the members are stalwarts in the Apache
> >communities, this should be a cinch IMO.
> >   =E2=80=95 The work on this new code can start, and depending on its s=
tate, and
> >assuming that the experiment succeeds, can be merged into trunk and late=
r
> >picked up in the next nearest & feasible Apache Hadoop release.
> >   =E2=80=95 While the work on forked-over-code goes on till we have a
> >functioning app, the current Apache Slider project continues to live,
> >with supporting releases etc. At some point in the future, when things
> >become clearer, more decisions can be taken on the (parts left behind in
> >the) incubating project=E2=80=99s future.
> >
> >Thoughts?
> >
> >Thanks
> >+Vinod
>
>

--001a113e237a9e8af40537f30e85--