Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Thu, 28 Aug 2014 20:29:08 +0000 (UTC)
From: "Siddharth Seth (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12737557.1409257627826.3234.1409257748311@arcas>
In-Reply-To: <JIRA.12737557.1409257627826@arcas>
References: <JIRA.12737557.1409257627826@arcas>
Subject: [jira] [Created] (YARN-2464) Provide Hadoop as a local resource (on
 HDFS) which can be used by other projcets
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Siddharth Seth created YARN-2464:
------------------------------------

             Summary: Provide Hadoop as a local resource (on HDFS) which can be used by other projcets
                 Key: YARN-2464
                 URL: https://issues.apache.org/jira/browse/YARN-2464
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Siddharth Seth


DEFAULT_YARN_APPLICATION_CLASSPATH are used by YARN projects to setup their AM / task classpaths if they have a dependency on Hadoop libraries.

It'll be useful to provide similar access to a Hadoop tarball (Hadoop libs, native libraries) etc, which could be used instead - for applications which do not want to rely upon Hadoop versions from a cluster node. This would also require functionality to update the classpath/env for the apps based on the structure of the tar.

As an example, MR has support for a full tar (for rolling upgrades). Similarly, Tez ships hadoop libraries along with it's build. I'm not sure about the Spark / Storm / HBase model for this - but using a common copy instead of everyone localizing Hadoop libraries would be useful.


--
This message was sent by Atlassian JIRA
(v6.2#6252)