hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
Date Fri, 01 Dec 2017 20:43:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274945#comment-16274945
] 

Hudson commented on MAPREDUCE-5124:
-----------------------------------

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13309 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13309/])
MAPREDUCE-5124. AM lacks flow control for task events. Contributed by (jlowe: rev 21d36273551fa45c4130e5523b6724358cf34b1e)
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestSpeculativeExecutionWithMRApp.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/event/TaskAttemptStatusUpdateEvent.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRClientService.java
* (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java


> AM lacks flow control for task events
> -------------------------------------
>
>                 Key: MAPREDUCE-5124
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.0.3-alpha, 0.23.5
>            Reporter: Jason Lowe
>            Assignee: Peter Bacsko
>             Fix For: 2.7.5, 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4
>
>         Attachments: MAPREDUCE-5124-001.patch, MAPREDUCE-5124-002.patch, MAPREDUCE-5124-003.patch,
MAPREDUCE-5124-CoalescingPOC-1.patch, MAPREDUCE-5124-CoalescingPOC2.patch, MAPREDUCE-5124-CoalescingPOC3.patch,
MAPREDUCE-5124-branch-2.001.patch, MAPREDUCE-5124-branch-2.002.patch, MAPREDUCE-5124-branch-2.7.001.patch,
MAPREDUCE-5124-branch-2.7.002.patch, MAPREDUCE-5124-branch-2.7.002.patch, MAPREDUCE-5124-branch-2.8.001.patch,
MAPREDUCE-5124-branch-2.8.001.patch, MAPREDUCE-5124-branch-2.8.002.patch, MAPREDUCE-5124-branch-2.8.002.patch,
MAPREDUCE-5124-branch-2.9.001.patch, MAPREDUCE-5124-branch-2.9.002.patch, MAPREDUCE-5124-proto.2.txt,
MAPREDUCE-5124-prototype.txt
>
>
> The AM does not have any flow control to limit the incoming rate of events from tasks.
 If the AM is unable to keep pace with the rate of incoming events for a sufficient period
of time then it will eventually exhaust the heap and crash.  MAPREDUCE-5043 addressed a major
bottleneck for event processing, but the AM could still get behind if it's starved for CPU
and/or handling a very large job with tens of thousands of active tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message