hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1151) Ability to configure auxiliary services from HDFS-based JAR files
Date Fri, 06 Sep 2013 11:33:53 GMT

    [ https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13760146#comment-13760146
] 

Steve Loughran commented on YARN-1151:
--------------------------------------

it gets more complex once you add in failure handling: how does the NM react to the aux service
process failing? How does the NM shut it down? where do the logs go? Who does it run as?

We effectively do have a system for doing this, it is called YARN. What sounds needed here
is a way to tell *something* that an NM has started and then give it the option of creating
and deploying a container on it.

That something should, obviously, be a YARN app itself, since they are set up to build up
command lines, copy in JARs, handle failures, etc. 

What we don't have is
# anything that starts a specific long-lived YARN AM service on cluster startup. 
# a way for an AM to list all the hosts and demand a container on every one, irrespective
of what is already there. (actually you could probably do it by asking for 0 RAM and vhosts,
but the min resource config options are designed to stop users doing this). YARN-1160 covers
that problem


                
> Ability to configure auxiliary services from HDFS-based JAR files
> -----------------------------------------------------------------
>
>                 Key: YARN-1151
>                 URL: https://issues.apache.org/jira/browse/YARN-1151
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: john lilley
>            Priority: Minor
>              Labels: auxiliary-service, yarn
>
> I would like to install an auxiliary service in Hadoop YARN without actually installing
files/services on every node in the system.  Discussions on the user@ list indicate that this
is not easily done.  The reason we want an auxiliary service is that our application has some
persistent-data components that are not appropriate for HDFS.  In fact, they are somewhat
analogous to the mapper output of MapReduce's shuffle, which is what led me to auxiliary-services
in the first place.  It would be much easier if we could just place our service's JARs in
HDFS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message