hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
Date Wed, 22 Nov 2017 22:09:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263439#comment-16263439

Jason Lowe commented on MAPREDUCE-5124:

Thanks for updating the patch!  Looks good overall.  I agree the unit test failure appears
to be unrelated as is the license issue.

For the Preconditions check that was added, the code ends up doing all the work of constructing
the message string only to throw it away most of the time.  The code should either do a straightforward
null-check-and-throw or use the Preconditions method that takes a format string and format
arguments so the string is only formatted if the check fails.

It would be good to fix the checkstyle indentation issues.

> AM lacks flow control for task events
> -------------------------------------
>                 Key: MAPREDUCE-5124
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.0.3-alpha, 0.23.5
>            Reporter: Jason Lowe
>            Assignee: Peter Bacsko
>         Attachments: MAPREDUCE-5124-001.patch, MAPREDUCE-5124-002.patch, MAPREDUCE-5124-CoalescingPOC-1.patch,
MAPREDUCE-5124-CoalescingPOC2.patch, MAPREDUCE-5124-CoalescingPOC3.patch, MAPREDUCE-5124-proto.2.txt,
> The AM does not have any flow control to limit the incoming rate of events from tasks.
 If the AM is unable to keep pace with the rate of incoming events for a sufficient period
of time then it will eventually exhaust the heap and crash.  MAPREDUCE-5043 addressed a major
bottleneck for event processing, but the AM could still get behind if it's starved for CPU
and/or handling a very large job with tens of thousands of active tasks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message