hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5487) In task processes, JobConf is unnecessarily loaded again in Limits
Date Mon, 16 Sep 2013 14:20:53 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768340#comment-13768340

Sandy Ryza commented on MAPREDUCE-5487:

bq. Shouldn't Limits.init(job) be called after adding the mapred config as resource?

This would diverge from the previous behavior, which uses a vanilla JobConf for loading the
Limits constants.

bq. Personally, I don't like constants that are not 'constants', that seems to be the case
of these limits.

Totally agree, though also think it should be a separate patch.
> In task processes, JobConf is unnecessarily loaded again in Limits
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-5487
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5487
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: performance, task
>    Affects Versions: 2.1.0-beta
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: MAPREDUCE-5487-1.patch, MAPREDUCE-5487.patch
> Limits statically loads a JobConf, which incurs costs of reading files from disk and
parsing XML.  The contents of this JobConf are identical to the one loaded by YarnChild (before
adding job.xml as a resource).  Allowing Limits to initialize with the JobConf loaded in YarnChild
would reduce task startup time.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message