hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "BELUGA BEHR (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-8789) Add BoundedQueue to AsyncDispatcher
Date Sat, 22 Sep 2018 19:43:00 GMT

     [ https://issues.apache.org/jira/browse/YARN-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

BELUGA BEHR updated YARN-8789:
    Attachment: YARN-8789.6.patch

> Add BoundedQueue to AsyncDispatcher
> -----------------------------------
>                 Key: YARN-8789
>                 URL: https://issues.apache.org/jira/browse/YARN-8789
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: applications
>    Affects Versions: 3.2.0
>            Reporter: BELUGA BEHR
>            Assignee: BELUGA BEHR
>            Priority: Major
>         Attachments: YARN-8789.1.patch, YARN-8789.2.patch, YARN-8789.3.patch, YARN-8789.4.patch,
YARN-8789.5.patch, YARN-8789.6.patch
> I recently came across a scenario where an MR ApplicationMaster was failing with an OOM
exception.  It had many thousands of Mappers and thousands of Reducers.  It was noted that
in the logging that the event-queue of {{AsyncDispatcher}} had a very large number of item
in it and was seemingly never decreasing.
> I started looking at the code and thought it could use some clean up, simplification,
and the ability to specify a bounded queue so that any incoming events are throttled until
they can be processed.  This will protect the ApplicationMaster from a flood of events.
> Logging Message:
> Size of event-queue is xxx

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message