hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4421) Remove dependency on deployed MR jars
Date Thu, 13 Sep 2012 16:38:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454989#comment-13454989
] 

Alejandro Abdelnur commented on MAPREDUCE-4421:
-----------------------------------------------

Arun - make sense, AMs configs should not pollute YARN config. 

Now building on this ...

We are aiming at a YARN deployment which does not know anything about AMs that run on it.
This means the cluster setup should not have AMs site.xml files deployed in it. This means
that the cluster setup should not have AMs JARs deployed in it.

For one off custom AMs, JARs and config would be all provided by the client on submission.

For AMs like MapReduce, DistributedShell and widely used AMs in a given cluster, their JARs
and config site.xml files would be in HDFS. Then the 'yarn' client script should support a
'-ampath=' (HDFS path to AM resources) option and/or a '-amname=' (logical name of the AM
resources, a new config file am-site.xml in the cluster would have this mapping for blessed
AMs, as  suggested in my prev comment). 

Thoughts?

                
> Remove dependency on deployed MR jars
> -------------------------------------
>
>                 Key: MAPREDUCE-4421
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 2.0.0-alpha
>            Reporter: Arun C Murthy
>            Assignee: Vinod Kumar Vavilapalli
>
> Currently MR AM depends on MR jars being deployed on all nodes via implicit dependency
on YARN_APPLICATION_CLASSPATH. 
> We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and, probably, just
rely on adding a shaded MR jar along with job.jar to the dist-cache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message