hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian He (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1121) RMStateStore should flush all pending store events before closing
Date Tue, 05 Nov 2013 19:05:17 GMT

     [ https://issues.apache.org/jira/browse/YARN-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jian He updated YARN-1121:

    Attachment: YARN-1121.6.patch

bq. There are 3 new booleans with 8 combinations possible between them
Each of them serves different purpose, also add comment in the code:
- drained:  indicates the dispatcher queue's events have been drained and processed.
- drainingStopNeeded(renames to drainEventsOnStop):  a configuration boolean which enables
or disables drain on stop functionality.
- drainingStop(renames to blockNewRequests): only for the purpose to block newly coming events
while draining to stop.

bq. Given that storing stuff will be over the network and slow,  why not have a wait notify
between this thread and the draining thread?
To do that, we may need to add things in dispatcher's runnable like "if(queueEmpty) notify",
and this is likely to be invoked in every normal execution of the dispatch while loop if queue
is empty, even it's not actually in stop phase, which may create more overhead, as this AsyncDispatcher
is used everywhere.

bq. DrainEventHandler sounds misleading. 
- renames DrainEventHandler to DropEventHandler

bq.The other thing we can do is take a count of the number of pending events to drain at service
For that, we change the new logic from blocking new events coming to the queue To process
a fixed number of events out of the queue, again we may need one more counter to indicate
how many events we have processed out of the queue.

Uploaded a new patch to address the comments

> RMStateStore should flush all pending store events before closing
> -----------------------------------------------------------------
>                 Key: YARN-1121
>                 URL: https://issues.apache.org/jira/browse/YARN-1121
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Bikas Saha
>            Assignee: Jian He
>             Fix For: 2.2.1
>         Attachments: YARN-1121.1.patch, YARN-1121.2.patch, YARN-1121.2.patch, YARN-1121.3.patch,
YARN-1121.4.patch, YARN-1121.5.patch, YARN-1121.6.patch
> on serviceStop it should wait for all internal pending events to drain before stopping.

This message was sent by Atlassian JIRA

View raw message