hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-1474) Make schedulers services
Date Tue, 20 May 2014 02:01:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002707#comment-14002707
] 

Karthik Kambatla edited comment on YARN-1474 at 5/20/14 2:01 AM:
-----------------------------------------------------------------

Thanks [~ozawa] for your patience with the reviews. I guess we can leave setRMContext as is.
And, let us handle the incompatible change to reinitialize in a separate JIRA.

On an HA cluster, I noticed that the scheduler threads (FS - updateThread, continuousSchedulingThread;
CS - asyncSchedulerThread) start on init() itself. Ideally, the threads should start only
on start(). I guess we should adopt a modified version of your earlier patch:
# From {{reinitialize()}}, move the part corresponding to {{if (!initialized)}} to serviceInit.
# Don't call {{reinitialize()}} in serviceInit or serviceStart.
# For the individual threads in the schedulers, init them in serviceInit, but call thread.start()
in serviceStart()
# serviceStop() for FS looks good. We should fix the serviceStop() for CS.

Other comments:
# In TestFairScheduler, the following is not required.
{code}
    // To initialize scheduler
    scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFairSchedulerEventLog, the following is not required. In this case and the above,
some tests might require calling resourceManager.startt().
{code}
    scheduler.serviceInit(conf);
    scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFifoScheduler, we don't need the following:
{code}
    scheduler.setRMContext(rm.getRMContext());
{code}
# TestFSLeafQueue doesn't need this either:
{code}
    scheduler.serviceInit(conf);
    scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestLeafQueue, we should call cs.init() instead of cs.serviceInit(). Also, in any other
places. 
# In TestQueueParsing, you might need to call capacityScheduler.init() in addition to or instead
of 
{code}
    capacityScheduler.reinitialize(conf, null);
{code}
# In TestRMContainerAllocator, we might have to call init() instead of reinitialize().
# In TestRMWebApp, we should call init() instead of reinitialize()

In general, in the tests,
# If there is an RM / Mock RM involved, we don't have to call setRMContext and reinitialize
as long as RM#init is called. 
# If there is no RM / Mock RM, we should call a setRMContext followed by init on the scheduler.
Subsequent, calls should remain reinitialize


was (Author: kkambatl):
Thanks [~ozawa] for your patience with the reviews. I guess we can leave setRMContext as is.
And, let us handle the incompatible change to reinitialize in a separate JIRA.

On an HA cluster, I noticed that the scheduler threads (FS - updateThread, continuousSchedulingThread;
CS - asyncSchedulerThread) start on init() itself. Ideally, the threads should start only
on start(). I guess we should adopt a modified version of your earlier patch:
# From {{reinitialize()}}, move the part corresponding to {{if (!initialized)}} to serviceInit.
# Don't call {{reinitialize()}} in serviceInit or serviceStart.
# For the individual threads in the schedulers, init them in serviceInit, but call thread.start()
in serviceStart()
# serviceStop() for FS looks good. We should fix the serviceStop() for CS.
# In TestFairScheduler, the following is not required.
{code}
    // To initialize scheduler
    scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFairSchedulerEventLog, the following is not required. In this case and the above,
some tests might require calling resourceManager.startt().
{code}
    scheduler.serviceInit(conf);
    scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestFifoScheduler, we don't need the following:
{code}
    scheduler.setRMContext(rm.getRMContext());
{code}
# TestFSLeafQueue doesn't need this either:
{code}
    scheduler.serviceInit(conf);
    scheduler.setRMContext(resourceManager.getRMContext());
{code}
# In TestLeafQueue, we should call cs.init() instead of cs.serviceInit(). Also, in any other
places. 
# In TestQueueParsing, you might need to call capacityScheduler.init() in addition to or instead
of 
{code}
    capacityScheduler.reinitialize(conf, null);
{code}
# In TestRMContainerAllocator, we might have to call init() instead of reinitialize().
# In TestRMWebApp, we should call init() instead of reinitialize()

In general, in the tests,
# If there is an RM / Mock RM involved, we don't have to call setRMContext and reinitialize
as long as RM#init is called. 
# If there is no RM / Mock RM, we should call a setRMContext followed by init on the scheduler.
Subsequent, calls should remain reinitialize

> Make schedulers services
> ------------------------
>
>                 Key: YARN-1474
>                 URL: https://issues.apache.org/jira/browse/YARN-1474
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>    Affects Versions: 2.3.0, 2.4.0
>            Reporter: Sandy Ryza
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-1474.1.patch, YARN-1474.10.patch, YARN-1474.11.patch, YARN-1474.12.patch,
YARN-1474.13.patch, YARN-1474.14.patch, YARN-1474.2.patch, YARN-1474.3.patch, YARN-1474.4.patch,
YARN-1474.5.patch, YARN-1474.6.patch, YARN-1474.7.patch, YARN-1474.8.patch, YARN-1474.9.patch
>
>
> Schedulers currently have a reinitialize but no start and stop.  Fitting them into the
YARN service model would make things more coherent.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message