hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1151) Ability to configure auxiliary services from HDFS-based JAR files
Date Fri, 23 Mar 2018 23:34:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412253#comment-16412253

Vinod Kumar Vavilapalli commented on YARN-1151:

>From the POC patch, it looks like NM will repeatedly download the tar again and again
on every restart, am I reading that right? We shouldn't be doing that. May be we should get
file-status, verify the checksum and skip if it is the same. Exactly like dist-cache. These
are some of the reasons why we should simply reuse the core ResourceLocalizationService for
localizing this.

We should also figure out the right behavior if HDFS is down. What should NM do on a fresh
start if HDFS is down? What it should do on a restart if HDFS is down?

> Ability to configure auxiliary services from HDFS-based JAR files
> -----------------------------------------------------------------
>                 Key: YARN-1151
>                 URL: https://issues.apache.org/jira/browse/YARN-1151
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.1.0-beta, 2.9.0
>            Reporter: john lilley
>            Assignee: Xuan Gong
>            Priority: Major
>              Labels: auxiliary-service, yarn
>         Attachments: YARN-1151.1.patch, YARN-1151.branch-2.poc.patch, [YARN-1151] [Design]
Configure auxiliary services from HDFS-based JAR files.pdf
> I would like to install an auxiliary service in Hadoop YARN without actually installing
files/services on every node in the system.  Discussions on the user@ list indicate that this
is not easily done.  The reason we want an auxiliary service is that our application has some
persistent-data components that are not appropriate for HDFS.  In fact, they are somewhat
analogous to the mapper output of MapReduce's shuffle, which is what led me to auxiliary-services
in the first place.  It would be much easier if we could just place our service's JARs in

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message