hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki ...@getopt.org>
Subject Re: Job scheduling (Re: Unable to run more than one job concurrently)
Date Fri, 19 May 2006 20:33:13 GMT
Doug Cutting wrote:
> Paul Sutter wrote:
>> (1) Allow submission times in the future, enabling the creation of
>> "background" jobs. My understanding is that job submission times are 
>> used to
>> prioritize scheduling. All tasks from a job submitted early run to
>> completion before those of a job submitted later. If we could submit any
>> days-long jobs with a submission time in the future, say the year 
>> 2010, and
>> any short hours-long jobs with the current time, that short job would be
>> able to interrupt the long job. Hack? Yes. Useful? I think so.
> I think this is equivalent to adding a job priority, where tasks with 
> the highest priority job are run first.  If jobs are at the same 
> priority, then the first submitted would run.  Adding priority would 
> add a bit more complexity, but would also be less of a hack.

Hmm.. If you compare it to a Unix scheduler, processes at the same 
priority have even chances of being run, regardless of which was started 
first - not only that, processes undergo a "priority decay", in that if 
they are running longer then their priority is lowered - this enables 
new processes to start quickly (and maybe quickly finish), and then 
fairly compete with other processes.

In our case, this would mean that jobs with the same priority would 
execute concurrently, sharing available map/reduce slots, and long 
running jobs would be gradually de-prioritized. This also means that the 
first job will slow down when the second one is started, but the second 
job will have a chance to make a good start (and perhaps quickly finish) 
and then, subject to the priority decay, run in parallel with other jobs 
(albeit slower) instead of being stuck in the wait queue.

And if the second job is started with a higher priority, it should 
preempt the first job (i.e. it should get proportionally more slots than 
the first job). If you need all cluster resources for a specific job, 
and don't want any other jobs to run, just set the priority to the 
highest value, thus preempting all other jobs (actually, it would 
suspend other already executing jobs, which would resume when your job 
is done - not a bad feature either!).

I think this is a relatively simple and well understood mechanism.

>> (2) Have a per-job total task count limit. Currently, we establish the
>> number of tasks each node runs, and how many map or reduce tasks we have
>> total in a given job. But it would be great if we could set a ceiling 
>> on the
>> number of tasks that run concurrently for a given job. This may help 
>> with
>> Andrzej's fetcher (since it is bandwidth constrained, maybe fewer 
>> concurrent
>> jobs would be fine?).
> I like this idea.  So if the highest-priority job is already running 
> at its task limit, then tasks can be run from the next 
> highest-priority job.  Should there be separate limits for maps and 
> reduces?

I like this idea too. I think a similar setting for the minimum number 
of tasks would be needed too? That would solve my problem. In fact, it 
would be probably better than the schema I described above, because it 
would guarantee certain minimum tasks running at any time.

This reminds me of the "idle time" and "real time" policies in BSD 
scheduler ... man 1 rtprio. The "real time" policy prevents the priority 
decay that normally occurs, and "idle time" policy allows processes to 
run only if CPU is idle.

Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

View raw message