hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gour Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5079) [Umbrella] Native YARN framework layer for services and beyond
Date Tue, 24 May 2016 20:11:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298805#comment-15298805
] 

Gour Saha commented on YARN-5079:
---------------------------------

[~kasha], thank you for your comments. Let me try to address few of the Slider-related questions.

bq. 1. For folks who don't know Slider all that well, it would help to enumerate the components
of Slider and a super brief description of what they do.

In one short sentence, Slider is a Universal Application Master. To add a little more, Slider
is a collection of tools and technologies to package, deploy, and manage services or long
running applications on Apache Hadoop YARN clusters. It is the fastest way to deploy any non-YARN-enabled
applications in a YARN cluster. It is comprised of 3 primary components - Slider AM, Slider
Client and Slider Agent. It also contains sample application packages, few written by the
Slider committers and others donated by the wider community.

For detailed documentation, one can start here:
http://slider.incubator.apache.org
http://slider.incubator.apache.org/design/architecture.html

bq. 2. What parts among these are being proposed for a merge?

The following is a proposal:
Components to be merged: Slider Core – comprising of AM and Client code, and the corresponding
tests that back them
Components that will not be merged: Slider Agent
Components that need more deliberation: Slider App-Packages

One way to do this, is by:
a. Create a branch in Hadoop and move Slider AM, Client and tests into it
b. Create a branch in Slider which will have only Agent and App-Packages and will point to
Slider core (AM and Client) as Hadoop modules

bq. 3. What happens to the rest of the parts? Continue to live in Slider? Is the slider community
comfortable with that? From the thread Gour pointed to, app packages seem like something that
need more discussion.

There will be a sufficiently large window, when the current Slider project & its releases
and the newly migrated code in YARN, will coexist. During this time, we will create v2 versions
of App-package definitions and find their final home in the future.

bq. 4. What release would we target for a Yarn-ified release of said Slider components? Hadoop
3?

It is likely that we start, by contributing this code into a branch in Hadoop and then work
on an Agent-less or thin-Agent architecture. Subsequently, we can include this in one of the
early Hadoop-3 releases.

bq. 6. With respect to CLI and UIs, how flexible is Slider and the Slider community with homogenizing
with Yarn where applicable. I doubt if we have a lot of this, but would be good to discuss.

Slider has an AM UI, which uses the YARN UI framework code and is already navigable from the
RM UI.

Currently, Slider has a separate CLI, but we can work towards homogenizing it.


> [Umbrella] Native YARN framework layer for services and beyond
> --------------------------------------------------------------
>
>                 Key: YARN-5079
>                 URL: https://issues.apache.org/jira/browse/YARN-5079
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>
> (See overview doc at YARN-4692, modifying and copy-pasting some of the relevant pieces
and sub-section 3.3.1 to track the specific sub-item.)
> (This is a companion to YARN-4793 in our effort to simplify the entire story, but focusing
on APIs)
> So far, YARN by design has restricted itself to having a very low-­level API that can
support any type of application. Frameworks like Apache Hadoop MapReduce, Apache Tez, Apache
Spark, Apache REEF, Apache Twill, Apache Helix and others ended up exposing higher level APIs
that end­-users can directly leverage to build their applications on top of YARN. On the
services side, Apache Slider has done something similar.
> With our current attention on making services first­-class and simplified, it's time
to take a fresh look at how we can make Apache Hadoop YARN support services well out of the
box. Beyond the functionality that I outlined in the previous sections in the doc on how NodeManagers
can be enhanced to help services, the biggest missing piece is the framework itself. There
is a lot of very important functionality that a services' framework can own together with
YARN in executing services end­-to­-end.
> In this JIRA I propose we look at having a native Apache Hadoop framework for running
services natively on YARN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message