hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3936) Clients should not enforce counter limits
Date Tue, 02 Oct 2012 18:23:08 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467935#comment-13467935

Robert Joseph Evans commented on MAPREDUCE-3936:

Under YARN the History Server also reads in the counters and that is where the counter limits
are needed in some form or another.  I don't really know what the correct solution is though.
  The history server cannot rely on the AM to enforce a limit because it is user code, so
if a job goes over the counter limits what should the history server do? Especially because
the client may be requesting data from the history server after the AM exits, so we really
want the history server to return results as close to what the AM did as possible.  Should
the history server refuse to load the job and throw an exception? That would result in the
client possibly failing, unless we are sure to update the client to deal with that eventuality
(I think this is the current behavior).  Should it just not load all of the counters? That
would result in odd behavior where the client would get a different view of the results when
talking to the history server instead of talking to the AM, but it wouldn't crash.
> Clients should not enforce counter limits 
> ------------------------------------------
>                 Key: MAPREDUCE-3936
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3936
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv1
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-3936.patch, MAPREDUCE-3936.patch
> The code for enforcing counter limits (from MAPREDUCE-1943) creates a static JobConf
instance to load the limits, which may throw an exception if the client limit is set to be
lower than the limit on the cluster (perhaps because the cluster limit was raised from the

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message