hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5124) AM lacks flow control for task events
Date Wed, 22 May 2013 07:55:21 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13663908#comment-13663908
] 

Tsuyoshi OZAWA commented on MAPREDUCE-5124:
-------------------------------------------

I apologize for delaying reply.

> So I would throw a ToBusyRetryLater type of exception for once time RPC calls when the
AsyncDispatcher's queue is over a high water mark, but for heartbeats I would want them to
scale the frequency based off of how busy the AsyncDispatcher is.

It's better idea. I'll try to implement both heartbeat-style RPC and one-time RPC.
                
> AM lacks flow control for task events
> -------------------------------------
>
>                 Key: MAPREDUCE-5124
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5124
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.0.3-alpha, 0.23.5
>            Reporter: Jason Lowe
>         Attachments: MAPREDUCE-5124-prototype.txt
>
>
> The AM does not have any flow control to limit the incoming rate of events from tasks.
 If the AM is unable to keep pace with the rate of incoming events for a sufficient period
of time then it will eventually exhaust the heap and crash.  MAPREDUCE-5043 addressed a major
bottleneck for event processing, but the AM could still get behind if it's starved for CPU
and/or handling a very large job with tens of thousands of active tasks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message