hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-702) minicluster classpath construction requires user to set yarn.is.minicluster in the job conf
Date Wed, 29 May 2013 22:58:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669850#comment-13669850
] 

Jonathan Hsieh commented on YARN-702:
-------------------------------------

The issue in hbase is that we'd like to have one single conf that gets the new settings but
the minicluster creates a new copy of the conf passed in instead of augmenting it. Thus when
hbase MR jobs run, we need to copy out a whole bunch of settings back out.  

Generally we create in this order:
* start mini zk
* start mini hdfs
* start mini hbase
* (optionally) start mini mr (and copy some settings over, also force isMiniCluster).

One workaround is to do a blanket copy of of the entire setup but that blanket action without
documentation understanding why we needed to do that seemed unacceptable.
                
> minicluster classpath construction requires user to set yarn.is.minicluster in the job
conf
> -------------------------------------------------------------------------------------------
>
>                 Key: YARN-702
>                 URL: https://issues.apache.org/jira/browse/YARN-702
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>
> YARN-129 improved classpath construction for miniclusters by, when yarn.is.minicluster
is set, adding the current JVM's classpath to the ContainerLaunchContext for the MR AM and
tasks.  An issue with this is that it requires the user to set yarn.is.minicluster on the
mapreduce side in the job conf, if they are not copying to RM conf into the jobconf.
> I think it would be better to bypass the ContainerLaunchContext and instead have the
nodemanager check the property, and if it is true, do the classpath additions there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message