hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vivek Ratan (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4513) Capacity scheduler should initialize tasks asynchronously
Date Tue, 04 Nov 2008 11:02:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644943#action_12644943
] 

Vivek Ratan commented on HADOOP-4513:
-------------------------------------

We should have a thread (call it the init thread) within the Capacity Scheduler that initializes
jobs (i.e., calls initTasks() on jobs even when the job is not ready to run, as per the scheduler).
The following features are desirable: 
* We want pre-initialized jobs in all the queues, since the Scheduler can possibly look at
all queues between consecutive runs of this thread. 
* Since initTasks() may take a fair bit of time for some jobs, we don't want to block on this
call as much as possible. 
* Remember that the init thread is really predicting which jobs the scheduler will run in
the next many heartbeats. It's hard to make this prediction as the decision of which job gets
a free slot depends on the state of the running jobs at that very instant (to compute user
quotas, for example). But there are some constraints that the thread can consider: 
** the waiting queue is sorted, and the scheduler considers jobs in that sort order. So the
init thread should pre-initialize jobs in the same order. 
** User quotas are tricky. Suppose a queue supports N concurrent users. The init thread should
really pre-initialize jobs from these N users only, as described earlier. But this list of
N users can change often, depending on which jobs from which users finish. So, to be safe,
the init thread should pre-initialize jobs from, say N+2 users, to be safe. 
* Since we're keeping the run-queue sorted (HADOOP-4471), and that jobs are moved from the
wait queue to the run queue once initTasks is called and the setup task is complete, the Scheduler
only needs to look at jobs in its run queue. If it doesn't find any job that can accept a
task, it moves on to the next queue. 
* If the init thread runs periodically, there may be a brief startup delay. Suppose the thread
runs every 5 seconds. Suppose a job is submitted to an empty queue just after the thread runs.
That job will not be initialized till the init thread runs next (a max delay of 5 secs). This
is not too bad, given that there will be a delay in running the setup task of that job too.
Worst case, the Scheduler can always cause the init thread to run as soon as a job is submitted
to an empty queue, but this may be an overkill. 

> Capacity scheduler should initialize tasks asynchronously
> ---------------------------------------------------------
>
>                 Key: HADOOP-4513
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4513
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/capacity-sched
>    Affects Versions: 0.19.0
>            Reporter: Hemanth Yamijala
>            Assignee: Sreekanth Ramakrishnan
>
> Currently, the capacity scheduler initializes tasks on demand, as opposed to the eager
initialization technique used by the default scheduler. This is done in order to save JT memory
footprint. However, the initialization is done in the {{assignTasks}} API which is not a good
idea as task initialization could be a time consuming operation. This JIRA is to move out
the initialization outside the {{assignTasks}} API and do it asynchronously.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message