hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3936) Clients should not enforce counter limits
Date Thu, 04 Oct 2012 21:13:48 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469708#comment-13469708

Luke Lu commented on MAPREDUCE-3936:

bq. are you saying that we should remove the enforcement of limits in AMs, since the user
could choose to do that by modifying the AM themselves?

Yes and no, as the limits can serve as a less dramatic way of prevent users from shooting
themselves in the foot than OOMs of AMs.

bq. Should it just not load all of the counters?

The current behavior in trunk (post 0.23) is actually fairly reasonable: it preserves all
the system counters and while make the exception queryable.

bq. if a job goes over the counter limits what should the history server do?

IMO, the best way to solve this problem is via MAPREDUCE-3061: the generic history server
can display the counters with existing behavior (preserving system counters while displaying
an error showing that user counter limits have been exceeded); an application specific history
server can display the counters however they want with the serialized counter json streamed
from the generic history server. 

> Clients should not enforce counter limits 
> ------------------------------------------
>                 Key: MAPREDUCE-3936
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3936
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv1
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-3936.patch, MAPREDUCE-3936.patch
> The code for enforcing counter limits (from MAPREDUCE-1943) creates a static JobConf
instance to load the limits, which may throw an exception if the client limit is set to be
lower than the limit on the cluster (perhaps because the cluster limit was raised from the

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message