hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-702) minicluster classpath construction requires user to set yarn.is.minicluster in the job conf
Date Thu, 30 May 2013 08:20:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670153#comment-13670153

Siddharth Seth commented on YARN-702:

bq. The issue in hbase is that we'd like to have one single conf that gets the new settings
but the minicluster creates a new copy of the conf passed in instead of augmenting it. Thus
when hbase MR jobs run, we need to copy out a whole bunch of settings back out.
There's some discussion about this on HBASE-7904. https://issues.apache.org/jira/browse/HBASE-7904?focusedCommentId=13611311&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611311

In the steps mentioned, is the same config object used ? Since MR is the last step, is it
possible to replace this configuration with the conf returned by the MiniMRCluster (getConfiguration()).

If the MiniMRCluster were to use the conf object passed in as a parameter, it would end up
adding a bunch of keys to this config (Everything defined in yarn-default and yarn-site) -
effectively the same as a getConfiguration on the cluster.

I think it's useful for the cluster to run with a different config, and not share this with
the multiple jobs that may be submitted to it. What may be possible is for the MiniMRCluster
to copy these configs into the user specified config. That would be very similar to a blanket

As mentioned before, I think it'll be useful to expose a limited configuration set (RM address,
isMiniCluster, etc) which can then be merged into client configs. Alternately, have the MiniCluster
merge in this limited set. This could be a start towards separating cluster specific configuration
and client specific configuration.
> minicluster classpath construction requires user to set yarn.is.minicluster in the job
> -------------------------------------------------------------------------------------------
>                 Key: YARN-702
>                 URL: https://issues.apache.org/jira/browse/YARN-702
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.0.4-alpha
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
> YARN-129 improved classpath construction for miniclusters by, when yarn.is.minicluster
is set, adding the current JVM's classpath to the ContainerLaunchContext for the MR AM and
tasks.  An issue with this is that it requires the user to set yarn.is.minicluster on the
mapreduce side in the job conf, if they are not copying to RM conf into the jobconf.
> I think it would be better to bypass the ContainerLaunchContext and instead have the
nodemanager check the property, and if it is true, do the classpath additions there.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message