hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5113) webhcat should allow configuring memory used by templetoncontroller map job in hadoop2
Date Tue, 03 Sep 2013 19:11:57 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756941#comment-13756941
] 

Eugene Koifman commented on HIVE-5113:
--------------------------------------

[~thejas] Some comments:
1. can templeton.controller.map.mem be named mapreduce.map.memory.mb, i.e. the same as the
Hadoop prop. I think it is easier to follow if prop are not renamed.  Or at least webhcat(templeton).mapreduce.map.memory.mb
if you are worried about namespace collisions.  (same for other new props)
2. The description in webhcat-default.xml:  Could it say that this is a Hadoop option and
is passed directly/as is to Templeton Controller MR job.  (In some cases default is specified
as '512' and in others '-Xmx300m').  It would tell users where to go get more doc that explains
what these props do.
3. templeton.controller.mr.child.opts is defined in the code but now webhcat-default.xml.
 Is that intentional?
4. public static final String HADOOP_MAP_JAVA_OPTS = "mapreduce.map.java.opts";
    public static final String HADOOP_MAP_MEMORY = "mapreduce.map.memory.mb";
    public static final String HADOOP_AM_MEMORY = "yarn.app.mapreduce.am.resource.mb";
    public static final String HADOOP_AM_JAVA_OPTS = "yarn.app.mapreduce.am.command-ops";
Are there symbolic constants in Hadoop code base for these?  Can they be used here?


                
> webhcat should allow configuring memory used by templetoncontroller map job in hadoop2
> --------------------------------------------------------------------------------------
>
>                 Key: HIVE-5113
>                 URL: https://issues.apache.org/jira/browse/HIVE-5113
>             Project: Hive
>          Issue Type: Improvement
>          Components: WebHCat
>            Reporter: Thejas M Nair
>            Assignee: Thejas M Nair
>         Attachments: HIVE-5113.1.patch
>
>
> Webhcat should allow the following hadoop2 config parameters to be set the templetoncontroller
map-only job that actually runs the pig/hive/mr command.
> mapreduce.map.memory.mb
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> It should also be set to reasonable defaults.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message