hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5079) [Umbrella] Native YARN framework layer for services and beyond
Date Wed, 03 Aug 2016 15:17:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406049#comment-15406049

Arun Suresh commented on YARN-5079:

We were going thru SLIDER-787. With regard to the state of slider as it is today, especially
with regard to upgarde, we had a couple of questions/clarifications:
# Every YARN Container started by the Slider AM consists of an Agent + the Component itself.
# Given 1) is true, I assume, the startContainers call to the NM would actually be starting
the Agent (as the root process) and the agent starts the Component process.
# Given 1) and 2), I assume component ‘upgrade’ is a yarn out-of-band protocol between
the slider AM and the Agent where the component process is brought down and restarted. This
also implies that currently the AM does not lose the allocation of the actual Container (unless
the Agent goes down)
# Is a Slider upgrade (AM + Agent) supported currently? I assume if only AM needs to be upgraded,
then work preserving AM restart must be enabled to allow the existing containers to bind to
the new AM. IF the Agent itself needs to upgraded, I guess there is no other option but to
lose the Container allocations (which I guess YARN-4876 might solve ?)
# From the docs, it also looks like the actual coordination / orchestration of the upgrade
is currently not the responsibility of Slider: An external tool can explicitly decide which
container (group of containers / roles) to upgrade, and can issue the necessary Slider commands.
# As per the discussions on this JIRA, the proposal is to NOT include the Slider agent in
the YARN merge. Is there a plan to provide a similar Agent like component within YARN? We
would still need an Agent like component that can (re)configure and (re)start processes as
well as doing fancy stuff like port allocations etc. Are you guys planning on opening up JIRAs
for the purpose (are they already there)?

> [Umbrella] Native YARN framework layer for services and beyond
> --------------------------------------------------------------
>                 Key: YARN-5079
>                 URL: https://issues.apache.org/jira/browse/YARN-5079
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
> (See overview doc at YARN-4692, modifying and copy-pasting some of the relevant pieces
and sub-section 3.3.1 to track the specific sub-item.)
> (This is a companion to YARN-4793 in our effort to simplify the entire story, but focusing
on APIs)
> So far, YARN by design has restricted itself to having a very low-­level API that can
support any type of application. Frameworks like Apache Hadoop MapReduce, Apache Tez, Apache
Spark, Apache REEF, Apache Twill, Apache Helix and others ended up exposing higher level APIs
that end­-users can directly leverage to build their applications on top of YARN. On the
services side, Apache Slider has done something similar.
> With our current attention on making services first­-class and simplified, it's time
to take a fresh look at how we can make Apache Hadoop YARN support services well out of the
box. Beyond the functionality that I outlined in the previous sections in the doc on how NodeManagers
can be enhanced to help services, the biggest missing piece is the framework itself. There
is a lot of very important functionality that a services' framework can own together with
YARN in executing services end­-to­-end.
> In this JIRA I propose we look at having a native Apache Hadoop framework for running
services natively on YARN.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message