hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5276) print more info when event queue is blocked
Date Fri, 24 Nov 2017 06:53:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16264987#comment-16264987
] 

Arun Suresh commented on YARN-5276:
-----------------------------------

Thanks [~sandflee] for raising this and for the patch.

Yup, it is definitely useful to find out exactly which events are backed up in the queue.
In our production clusters, we do have a patch this is similar to what you have put up. One
suggestion though: instead of iterating an creating a new map everytime (not to mention the
performance cost of iterating thru the queue everytime), what we do is to keep a ConcurrentHashMap<EventType,
Counter> as a persistent field in the Dispatcher. We then update the counter (or insert/remove
an entry if required) everytime an event is inserted or removed from the queue. And we just
dump the map if the total size of the queue is greater than some value. 

Additionally, it would also be useful to dump the last event that was dispatched - this can
give a hint of which handler potentially caused the backup

> print more info when event queue is blocked
> -------------------------------------------
>
>                 Key: YARN-5276
>                 URL: https://issues.apache.org/jira/browse/YARN-5276
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager
>            Reporter: sandflee
>            Assignee: sandflee
>              Labels: oct16-easy
>         Attachments: YARN-5276.01.patch, YARN-5276.02.patch, YARN-5276.03.patch, YARN-5276.04.patch
>
>
> we now see logs like "Size of event-queue is 498000, Size of event-queue is 499000" and
difficult to know which event flood the queue. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message