hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "john lilley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1151) Ability to configure auxiliary services from HDFS-based JAR files
Date Thu, 05 Sep 2013 21:17:51 GMT

    [ https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13759475#comment-13759475
] 

john lilley commented on YARN-1151:
-----------------------------------

I didn't realize that auxiliary-services were in the NM address space; I had assumed they
would be separate processes.  What I am really looking for is just some way to start a long-running
service that meets these criteria:
1) Mirrors the NM lifetime.
2) Doesn't have to be installed on every node manually.
3) (probably implied by 2) can be staged on HDFS.
Perhaps instead of improvements to the existing "auxiliary service" this would simply be a
command that was run by the NM on startup, and stopped by the NM on shutdown.  The command
could be anything, similar to YARN tasks.

On further thought, it is not strictly necessary for the "long running service" to start and
stop with the NM for this purpose.  It could be the AM that requests its availability, which
would trigger the NM to start it before any containers are created.

                
> Ability to configure auxiliary services from HDFS-based JAR files
> -----------------------------------------------------------------
>
>                 Key: YARN-1151
>                 URL: https://issues.apache.org/jira/browse/YARN-1151
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: john lilley
>            Priority: Minor
>              Labels: auxiliary-service, yarn
>
> I would like to install an auxiliary service in Hadoop YARN without actually installing
files/services on every node in the system.  Discussions on the user@ list indicate that this
is not easily done.  The reason we want an auxiliary service is that our application has some
persistent-data components that are not appropriate for HDFS.  In fact, they are somewhat
analogous to the mapper output of MapReduce's shuffle, which is what led me to auxiliary-services
in the first place.  It would be much easier if we could just place our service's JARs in
HDFS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message